Force Feedback System Including Multi-Tasking Graphical Host Environment

ABSTRACT

A force feedback system provides components for use in a force feedback system including a host computer and a force feedback interface device. An architecture for a host computer allows multi-tasking application programs to interface with the force feedback device without conflicts. One embodiment of a force feedback device provides both relative position reporting and absolute position reporting to allow great flexibility. A different device embodiment provides relative position reporting device allowing maximum compatibility with existing software. Information such as ballistic parameters and screen size sent from the host to the force feedback device allow accurate mouse positions and graphical object positions to be determined in the force feedback environment. Force feedback effects and structures are further described, such as events and enclosures.

CROSS-REFERENCES TO RELATED APPLICATION

This application is a continuation of U.S. application Ser. No.11/824,593, filed Jun. 28, 2007, now U.S. Pat. No. 8,020,095, issued onSep. 13, 2011, and entitled “Force Feedback System IncludingMulti-Tasking Graphical Host Environment,” which is acontinuation-in-part of U.S. application Ser. No. 11/504,131 filed onAug. 14, 2006, and entitled “Force Feedback System IncludingMulti-Tasking Graphical Host Environment and Interface Device” which isa continuation of U.S. application Ser. No. 09/974,197 filed on Oct.9,2001, now U.S. Pat. No. 7,168,042, issued on Jan. 23, 2007, andentitled, “Force Effects for Object Types in a Graphical User Interface”which is a continuation of U.S. application Ser. No. 08/970,953, filedNov. 14, 1997, now U.S. Pat. No. 6,300,936, issued on Oct. 9, 2001, andentitled “Force Feedback System Including Multi-Tasking Graphical HostEnvironment and Interface Device,” in the names of the same inventorsand commonly owned herewith, the entirety of all of which are herebyincorporated by reference.

STATEMENT OF GOVERNMENT RIGHTS

One or more claimed embodiments described herein made with Governmentsupport under Contract Number F41624-96-C-6029, awarded by theDepartment of Defense. The Government has certain rights herein.

BACKGROUND

One or more embodiments described herein relates generally to interfacedevices for allowing humans to interface with computer systems, and moreparticularly to computer interface devices that allow the user toprovide input to computer systems and provide force feedback to theuser.

Computer systems are used extensively to implement many applications,such as word processing, data management, simulations, games, and othertasks. A computer system typically displays a visual environment to auser on a display screen or other visual output device. Users caninteract with the displayed environment to perform functions on thecomputer, play a game, experience a simulated environment, use acomputer aided design (CAD) system, etc. One visual environment that isparticularly common is a graphical user interface (GUI). GUIs presentvisual images which describe various graphical metaphors of a program oroperating system implemented on the computer. Common operating systemsusing GUIs include the Windows.™. operating system from MicrosoftCorporation and the MacOS operating system from Apple Computer, Inc. Theuser typically moves a displayed, user-controlled graphical object, suchas a graphical object or pointer, across a computer screen and ontoother displayed graphical objects or predefined screen regions, and theninputs a command to execute a given selection or operation. The objectsor regions (“targets”) can include, for example, icons, windows,pull-down menus, buttons, and scroll bars. Most GUIs are currently2-dimensional as displayed on a computer screen; however, threedimensional (3-D) GUIs that present simulated 3-D environments on a 2-Dscreen can also be provided. Other programs or environments that mayprovide user-controlled graphical objects such as a graphical object ora “view” controlled by the user include graphical “web pages” or otherenvironments offered on the World Wide Web of the Internet, CADprograms, video games, virtual reality simulations, etc.

The user interaction with, and manipulation of, the computer environmentis achieved using any of a variety of types of human-computer interfacedevices that are connected to the computer system controlling thedisplayed environment. Force feedback interface devices allow a user toexperience forces on the manipulated user object based on interactionsand events within the displayed graphical environment. Typically,computer-controlled actuators are used to output forces on the userobject to simulate various sensations, such as an obstruction force whenmoving the graphical object into a wall, a damping force to resistmotion of the graphical object, and a spring force to bias the graphicalobject to move back toward a starting position of the spring.

When implementing force feedback sensations in a GUI of an operatingsystem, several problems can arise. One problem is the use of forcefeedback when multiple application programs are simultaneously runningin a multi-tasking environment on the host computer. Most operatingsystems allow such multi-tasking, for example, to allow a user tointeract with one application while one or more applications are alsorunning, receiving data, outputting data, or performing other tasks. Forexample, in the Windows.™. operating system, one application is the“active” application that typically displays an active window in theGUI. The user can manipulate the functions of the active applicationusing the graphical object. Other inactive applications are also runningand may have inactive windows displayed in the GUI. The user can switchto a different application by clicking the graphical object in aninactive window, for example, which causes the new application to be theactive application and the formerly active application to becomeinactive.

Each application run by an operating system may have its own set offorce sensations that it needs to command to the force feedback device.Thus, one application may need to command spring, force, vibration, andtexture force sensations, while a different application may need tocommand spring, damper, and jolt force sensations. The force feedbackdevice typically cannot store all possible force sensations for eachapplication running in the operating system, so there is a problem ofwhich force sensations the force feedback device should store andimplement at any one time. In addition, if two of the multi-taskingapplications command conflicting force sensations, the force feedbackdevice needs to choose one of the force sensations to output. Currentlyis no system or method of doing so.

A different problem occurs when using a force feedback device with aGUI. Traditional mouse controllers used with GUIs are relative positionreporting devices, i.e., they report only changes in position of themouse to the host computer, which the host computer uses to calculate anew position for the graphical object on the screen. Many force feedbackdevices, in contrast, are typically absolute position reporting deviceswhich report an absolute position of the graphical object, such asscreen coordinates, to the host computer. This is because the forcefeedback device needs to know the graphical object position toaccurately determine when forces are to be applied and to accuratelycalculate the forces. However, it would be desirable in some instancesto have a relative position reporting force feedback device, since thehost computer is standardized to receive and interpret relativepositions at the most basic level. Furthermore, such a relative devicewould permit the host computer to perform needed adjustments tographical object position, such as ballistics calculations which modifygraphical object position based on mouse velocity to provide enhancedcontrol. If the host computer performs such adjustments, the forcefeedback device processors are relieved of computational burden.

Another problem occurs when force feedback is implemented with a GUI orother graphical environment and the graphical environment changesresolution or aspect ratio. For example, if a resolution of640.times.480 is being displayed by the host computer on a screen, theforce feedback device assumes that graphical objects in the GUI have asize proportional to screen dimensions and outputs forces accordingly.However, when the resolution is changed, the objects displayed on thescreen change size in proportion to the screen dimensions. The forcefeedback device continues to check for conditions and generate forces asif the old resolution were active, resulting in forces that do notcorrelate with displayed interactions on the screen. The aspect ratio ofa display screen can also change, e.g., when two screens are used toprovide double the amount of displayed area in the GUI, the aspect ratiodoubles in one dimension. Using prior art force feedback devices, forcescan become distorted from such an aspect ratio change. For example, acircle object displayed on the screen may have forces at its bordersthat feel like an ellipse to the user of the interface device, since theaspect ratios of the screen and the mouse workspace are different.

SUMMARY

A force feedback system provides components for use in a force feedbacksystem including a host computer and a force feedback interface device.An architecture for a host computer allows multi-tasking applicationprograms to interface with the force feedback device without conflicts.One embodiment of a force feedback device provides both relativeposition reporting and absolute position reporting to allow greatflexibility. A different device embodiment provides relative positionreporting device allowing maximum compatibility with existing software.Information such as ballistic parameters and screen size sent from thehost to the force feedback device allow accurate mouse positions andgraphical object positions to be determined in the force feedbackenvironment. Force feedback effects and structures are furtherdescribed, such as events and enclosures. These and other advantageswill become apparent to those skilled in the art upon a reading of thefollowing specification and a study of the several figures of thedrawing.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated into and constitute apart of this specification, illustrate one or more examples ofembodiments and, together with the description of example embodiments,serve to explain the principles and implementations of the embodiments.

FIG. 1 a is a perspective view of a mouse-based user device inaccordance with an embodiment;

FIG. 1 b is a perspective view of a mechanism of the mouse-based userdevice in accordance with an embodiment;

FIG. 2 is a perspective view of a touch panel-based user device inaccordance with an embodiment;

FIG. 3 is a block diagram of the system in accordance with anembodiment;

FIG. 4 is a block diagram of an architecture for a host computerproviding multiple application programs communicating with the forcefeedback device in accordance with an embodiment;

FIG. 5 is a diagrammatic illustration of a background applicationprogram control panel allowing a user to characterize background forcesin accordance with an embodiment;

FIGS. 6 a, 6 b, and 6 c are diagrammatic illustrations of embodiments ofan enclosure force effect;

FIG. 7 is a diagrammatic illustration of a texture force effect;

FIG. 8 is a diagrammatic illustration of a grid force effect;

FIG. 9 is a block diagram illustrating implementing the force feedbackdevice and system in accordance with an embodiment;

FIG. 10 is a flow diagram illustrating a method of implementing positionreporting in accordance with an embodiment;

FIG. 11 is a flow diagram illustrating a method of determining graphicalobject position and indexing in accordance with an embodiment;

FIG. 12 is a block diagram illustrating implementing the force feedbackdevice and system in accordance with an embodiment;

FIG. 13 is a flow diagram illustrating a method of implementing positionreporting in accordance with an embodiment; and

FIGS. 14 and 15 are diagrammatic illustrations showing the errorcorrection between graphical object positions from host computer andforce feedback device in accordance with an embodiment.

DETAILED DESCRIPTION

Example embodiments are described herein in the context of a system ofcomputers, servers, and software. Those of ordinary skill in the artwill realize that the following description is illustrative only and isnot intended to be in any way limiting. Other embodiments will readilysuggest themselves to such skilled persons having the benefit of thisdisclosure. Reference will now be made in detail to implementations ofthe example embodiments as illustrated in the accompanying drawings. Thesame reference indicators will be used throughout the drawings and thefollowing description to refer to the same or like items.

In the interest of clarity, not all of the routine features of theimplementations described herein are shown and described. It will, ofcourse, be appreciated that in the development of any such actualimplementation, numerous implementation-specific decisions must be madein order to achieve the developer's specific goals, such as compliancewith application- and business-related constraints, and that thesespecific goals will vary from one implementation to another and from onedeveloper to another. Moreover, it will be appreciated that such adevelopment effort might be complex and time-consuming, but wouldnevertheless be a routine undertaking of engineering for those ofordinary skill in the art having the benefit of this disclosure.

In accordance with this disclosure, the components, process steps,and/or data structures described herein may be implemented using varioustypes of operating systems, computing platforms, computer programs,and/or general purpose machines. In addition, those of ordinary skill inthe art will recognize that devices of a less general purpose nature,such as hardwired devices, field programmable gate arrays (FPGAs),application specific integrated circuits (ASICs), or the like, may alsobe used without departing from the scope and spirit of the inventiveconcepts disclosed herein. It is understood that the phrase “anembodiment” encompasses more than one embodiment and is thus not limitedto only one embodiment. Where a method comprising a series of processsteps is implemented by a computer or a machine and those process stepscan be stored as a series of instructions readable by the machine, theymay be stored on a tangible medium such as a computer memory device(e.g., ROM (Read Only Memory), PROM (Programmable Read Only Memory),EEPROM (Electrically Erasable Programmable Read Only Memory), FLASHMemory, Jump Drive, and the like), magnetic storage medium (e.g., tape,magnetic disk drive, and the like), optical storage medium (e.g.,CD-ROM, DVD-ROM, paper card, paper tape and the like) and other types ofprogram memory.

FIG. 1 a is a perspective view of a force feedback mouse interfacesystem 10 in accordance with an embodiment. The system 10 is capable ofproviding input from the user to a host computer based on the user'smanipulation of the mouse and capable of providing force feedback to theuser of the mouse system based on events occurring in an environmentimplemented by the host computer. Mouse system 10 includes an interfacedevice 11 that includes a user manipulatable object or manipulandum 12and an interface 14, and a host computer 18.

User object (or “manipulandum”) 12 is a physical object that ispreferably grasped, touched, gripped and/or manipulated by a user. Forexample, images are displayed and/or modified on a display screen 20 ofthe computer system 18 in response to such manipulations. In anembodiment, the user object 12 is a mouse 12 that is shaped so that auser's fingers or hand may comfortably grasp the object and move it inthe provided degrees of freedom in physical space. For example, a usercan move mouse 12 to correspondingly move a computer generated graphicalobject in a graphical environment provided by computer 18. It should benoted that the term graphical object encompasses a variety of objectswhich can be controlled within the graphical environment. The graphicalobject includes, but is not limited to, a cursor, an avatar, a characteror vehicle in a video game or virtual environment, an icon, a menu item,a virtual tool (i.e., medical training simulation) and/or amanipulatable object (e.g., volume slider, virtual map). The availabledegrees of freedom in which mouse 12 can be moved are determined fromthe interface 14, described below. The mouse 12 may include one or morebuttons 15 to allow the user to provide additional commands to thecomputer system. In an embodiment, the mouse 12 does not include anybuttons.

It will be appreciated that a number of other types of user manipulableobjects fall within the definition of the user object. In exemplaryembodiments, such objects may include a sphere; a trackball; a puck; ajoystick; cubical- or other-shaped hand grips; a fingertip receptaclefor receiving a finger or a stylus; a flat, planar surface like aplastic card having a rubberized, contoured, and/or bumpy surface; ahandheld video game controller; a handheld remote control used forcontrolling web pages or other devices; a touch panel; a touch pad;and/or a touch screen.

In an embodiment, a small fingertip joystick can be provided, where asmall stick is moved in a small planar region with a user's fingertips.Spring forces can be provided by the actuators of the device 11 to biasthe stick (or any type of joystick) toward the center of the planarregion to simulate a spring return on the joystick. Since fingertips areused, output forces need not be as high a magnitude as in otherembodiments. Also, mouse 12 can be provided with such a centering springbias, e.g., when used like a joystick in gaming applications.

Interface 14 interfaces mechanical and electrical input and outputbetween the mouse 12 and host computer 18 implementing the applicationprogram, such as a GUI, simulation or game environment. Interface 14provides multiple degrees of freedom to mouse 12, and in an embodiment,two linear, planar degrees of freedom are provided to the mouse, asshown by arrows 22. In other embodiments, greater or fewer degrees offreedom can be provided, as well as rotary degrees of freedom. For manyapplications, mouse 12 need only be moved in a very small workspacearea.

In an embodiment, the user manipulates mouse 12 in a planar workspace,much like a traditional mouse, and the position of mouse 12 istranslated into a form suitable for interpretation by position sensorsof the interface 14. The sensors track the movement of the mouse 12 inplanar space and provide suitable electronic signals to an electronicportion of interface 14. The interface 14 provides position informationto host computer 18. In addition, host computer 18 and/or interface 14provide force feedback signals to actuators coupled to interface 14, andthe actuators generate forces on members of the mechanical portion ofthe interface 14 to provide forces on mouse 12 in provided or desireddegrees of freedom. The user experiences the forces generated on themouse 12 as realistic simulations of force sensations such as jolts,springs, textures, enclosures, circles, ellipses, grids, vibrations,“barrier” forces, and the like.

The electronic portion of interface 14 may couple the mechanical portion24 of the interface to the host computer 18. The electronic portion ispreferably included within the housing 21 of the interface 14 or,alternatively, the electronic portion may be included in host computer18 or as a separate unit with its own housing. More particularly,interface 14 includes a local microprocessor distinct and separate fromany microprocessors in the host computer 18 to control force feedback onmouse 12 independently of the host computer, as well as sensor andactuator interfaces that convert electrical signals to appropriate formsusable by the mechanical portion of interface 14 and host computer 18.

For example, a rigid surface is generated on a computer screen 20 and acomputer object (e.g., cursor) controlled by the user collides with thesurface. In an embodiment, high-level host commands can be used toprovide the various forces associated with the rigid surface. The localcontrol mode using a local microprocessor in interface 14 may be helpfulin increasing the response time for forces applied to the user object,which is essential in creating realistic and accurate force feedback.For example, a host computer 18 may send a “spatial representation” tothe local microprocessor, which is data describing the locations of someor all the graphical objects displayed in a GUI or other graphicalenvironment which are associated with forces and thetypes/characteristics of these graphical objects. The microprocessor canstore such a spatial representation in a local memory, and thus will beable to determine interactions between the user object and graphicalobjects (such as the rigid surface) independently of the host computer.In addition, the microprocessor can be provided with the necessaryinstructions or data to check sensor readings, determine cursor andtarget positions, and determine output forces independently of hostcomputer 18. The host could implement program functions (such asdisplaying images) when appropriate, and synchronization commands can becommunicated between the microprocessor and the host 18 to correlate themicroprocessor and host processes. Such commands and relatedfunctionality is discussed in greater detail below. In an embodiment,the computer 18 can directly send force feedback signals to theinterface 14 to generate forces on mouse 12. A suitable embodiment ofthe electrical portion of interface 14 is described in detail withreference to FIG. 3, although the description may apply to otherembodiments.

The interface 14 can be coupled to the computer 18 by a bus 17, whichcommunicates signals between interface 14 and computer 18. In anembodiment, the computer 18 provides power to the interface 14 (e.g.,when bus 17 includes a USB interface). In an embodiment, signals can besent between interface 14 and computer 18 by wirelesstransmission/reception. In an embodiment, the interface 14 serves as aninput/output (I/O) device for the computer 18. The interface 14 can alsoreceive inputs from other input devices or controls that are associatedwith mouse system 10 and can relay those inputs to computer 18. Forexample, commands sent by the user activating a button on mouse 12 canbe relayed to computer 18 by interface 14 to implement a command orcause the computer 18 to output a command to the interface 14.

Host computer 18 is a personal computer or workstation in an embodiment.For example, the computer 18 can operate under the Windows.™. or MS-DOSoperating system. In an embodiment, the host computer system 18 may beone of a variety of home video game systems, such as systems availablefrom Nintendo, Sega, or Sony. In other embodiments, host computer system18 can be a “set top box” which can be used, for example, to provideinteractive television functions to users, or a “network-” or“internet-computer” which allows users to interact with a local orglobal network using standard connections and protocols such as used forthe Internet and World Wide Web. The host computer may be a standalonedevice such as an automated teller machine (ATM), kiosk, point of sale(POS) device found at a grocery market or department store, PersonalDigital Assistant (PDA), and/or mobile phone. Host computer preferablyincludes a host microprocessor, random access memory (RAM), read onlymemory (ROM), input/output (I/O) circuitry, and other components ofcomputers well-known to those skilled in the art.

The host computer 18 implements and/or performs one or more applicationprograms (“applications”) with which a user is interacting via the userobject and/or other peripherals, if appropriate, and which can includeforce feedback functionality. For example, the host application programscan include a simulation, video game, Web page or browser thatimplements HTML or VRML instructions, word processor, drawing program,spreadsheet, scientific analysis program, or other application programthat utilizes input of the user device and outputs force feedbackcommands to the user device.

In an embodiment, an operating system such as Windows.™., MS-DOS, MacOS,Unix, is also running on the host computer and includes its own forcefeedback capabilities. In an embodiment, the operating system andapplication programs utilize a graphical user interface (GUI) to presentoptions to a user, display data and images, and receive input from theuser. In the preferred embodiment, multiple applications can runsimultaneously in a multitasking environment of the host computer, as isdetailed below. Herein, computer 18 may be referred as displaying“graphical objects” or “computer objects.” These objects are notphysical objects, but are logical software unit collections of dataand/or procedures that may be displayed as images by computer 18 ondisplay screen 20, as is well-known to those skilled in the art. Thehost application program checks for input signals received from theelectronics and sensors of interface 14, and outputs force values and/orcommands to be converted into forces on the user device.

Display device 20 can be included in the host computer 18 and can be astandard display screen (LCD, CRT, etc.), 3-D goggles, or any othervisual output device. In an embodiment, the host application providesimages to be displayed on display device 20 and/or other feedback, suchas auditory signals. For example, display screen 20 can display imagesfrom a GUI. Images describing a moving, first-person point-of-view canbe displayed, as in a virtual reality game. Or, images describing athird-person perspective of objects, backgrounds, etc. can be displayed.Alternatively, images from a simulation, such as a medical simulation,can be displayed (e.g., images of tissue and a representation of amanipulated user object 12 moving through the tissue).

There are two primary “control paradigms” of operation for the userdevice: position control and rate control. Position control refers to amapping of the input of the user device in which displacement ormovement of the user device in a physical space directly dictatesdisplacement of a graphical object. Under a position control mapping,the graphical object does not move unless the user object is in motion.Also, “ballistics” or other non-linear adjustments to cursor positioncan be used in which, for example, slow motions of the user device havea different scaling factor for graphical object movement than fastmotions of the user device. This allows more control of short movementsof the graphical object in the graphical environment. Several differentways of implementing ballistics and other control adjustments in a forcefeedback device are described in U.S. Pat. No. 6,252,579, and theseadjustments can be used if desired.

As shown in FIG. 1 a, the host computer may have its own “screen frame”28 (or host frame) which is displayed on the display screen 20. Incontrast, the user device has its own “device frame” (or local frame) 30in which the user device is capable of moving. In a position controlparadigm, the position (or change in position) of the graphical objectin the host frame 30 corresponds to a position (or change in position)of the user device in the local frame 28.

Rate control is also used as a control paradigm. This refers to amapping in which the displacement of the user device is abstractlymapped to motion of the graphical object under control. There is not adirect physical mapping between physical object (mouse) motion andcomputer object motion. Thus, most rate control paradigms arefundamentally different from position control in that the user objectcan be held steady at a given position but the controlled computerobject is in motion at a commanded or given velocity, while the positioncontrol paradigm only allows the controlled computer object to be inmotion if the user object is in motion.

The interface system 10 is useful for both position control (“isotonic”)tasks and rate control (“isometric”) tasks. In an example embodiment,the position of a mouse 12 in its local frame 30 workspace can bedirectly mapped to a position of a graphical object in the host frame 28on the display screen 20 in a position control paradigm. Alternatively,the displacement of the mouse 12 in a particular direction against anopposing output force can command the rate control tasks in an isometricmode.

In the embodiment shown in FIG. 1 a, the mouse 12 is supported andsuspended above a grounded pad 32 by the mechanical portion of interface14. Pad 32 or similar surface is supported by grounded surface 34. Pad32 (or alternatively grounded surface 34) provides additional supportfor the mouse and relieves stress on the mechanical portion of interface14. In addition, a wheel, roller, Teflon pad or other device can be usedto support the mouse. The user can move the mouse in a 2D planarworkspace to move the graphical object or objects in the GUI or performother tasks. In other graphical environments, such as a virtual realityvideo game, a user can be controlling a computer player or vehicle inthe virtual environment by manipulating the mouse 12. The computersystem tracks the position of the mouse 12 with sensors as the usermoves it. The computer system may also provide force feedback commandsto the mouse 12, for example, when the user moves the graphical objectagainst a generated surface such as an edge of a window, a virtual wall,etc. It thus appears and feels to the user that the mouse and thegraphical object are contacting real surfaces.

In an embodiment, the system 10 may also include an indexing function or“indexing mode” which allows the user to redefine the offset between thepositions of the mouse 12 or other user device in the local frame andthe graphical object in the host frame displayed by host computer 18. Inone implementation, the user may reposition the mouse 12 withoutproviding any input to the host computer, thus allowing the user toredefine the offset between the mouse's position and the graphicalobject's position. Such indexing is achieved through an input devicesuch as a button, switches, sensors, or other input devices. As long asthe indexing device is activated, the mouse 12 or other user device willbe in indexing mode. When the button is released (or indexing modeotherwise exited), the position of the cursor is again controlled by theuser device. In an embodiment, the mouse 12 includes a hand weightswitch which inherently causes indexing when the user removes her handor finger weight from the mouse 12. A different way to allow indexing isto provide a combined position control and rate control device whichallows different forms of control of the graphical object depending onthe position of the user device in its workspace. It should be notedthat the above description may be applied to all types of user devicesdescribed above and is therefore not limited to a mouse device.

FIG. 1 b is a perspective view of a mouse device 11 with the coverportion of housing 21 and the grounded pad 32 removed in accordance withan embodiment. Mechanical linkage 40 provides support for mouse 12 andcouples the mouse to a grounded surface 34, such as a tabletop or othersupport. In an embodiment, the linkage 40 is a 5-member (or “5-bar”)linkage. Fewer or greater numbers of members in the linkage can beprovided in alternate embodiments.

In an embodiment, the ground member 42 of the linkage 40 is a base forthe support of the linkage and is coupled to or resting on a groundsurface 34. The ground member 42 in FIG. 1 b is shown as a plate or basethat extends under mouse 12. The members of linkage 40 are rotatablycoupled to one another through the use of rotatable pivots or bearingassemblies, all referred to as “bearings” herein. Base member 44 isrotatably coupled to ground member 42 by a grounded bearing 52 and canrotate about an axis A. Link member 46 is rotatably coupled to basemember 44 by bearing 54 and can rotate about a floating axis B, and basemember 48 is rotatably coupled to ground member 42 by bearing 52 and canrotate about axis A. Link member 50 is rotatably coupled to base member48 by bearing 56 and can rotate about floating axis C, and link member50 is also rotatably coupled to link member 46 by bearing 58 such thatlink member 50 and link member 46 may rotate relative to each otherabout floating axis D.

Linkage 40 is formed as a five-member closed-loop chain. Each member inthe chain is rotatably coupled to two other members of the chain toprovide mouse 12 with two degrees of freedom, i.e., mouse 12 can bemoved within a planar x-y workspace. Mouse 12 is coupled to link members46 and 50 by rotary bearing 58 and may rotate at least partially aboutaxis D. A pad or other support can be provided under mouse 12 to helpsupport the mouse.

In an embodiment, a transducer system is used to sense the position ofmouse 12 in its workspace and to generate forces on the mouse 12. In anembodiment, the transducer system includes sensors 62 and actuators 64.The sensors 62 collectively sense the movement of the mouse 12 in theprovided degrees of freedom and send appropriate signals to theelectronic portion of interface 14. Sensor 62 a senses movement of linkmember 48 about axis A, and sensor 62 b senses movement of base member44 about axis A. These sensed positions about axis A allow thedetermination of the position of mouse 12 using known constants such asthe lengths of the members of linkage 40 and using well-known coordinatetransformations.

In an embodiment, the sensors 62 are grounded optical encoders thatsense the intermittent blockage of an emitted beam. A groundedemitter/detector portion 71 includes an emitter that emits a beam whichis detected by a grounded detector. A moving encoder disk portion or“arc” 74 is provided at the end of members 44 and 48 which each blockthe beam for the respective sensor in predetermined spatial incrementsand allows a processor to determine the position of the arc 74 and thus,the members 44 and 48, by counting the spatial increments. Also, avelocity of members 44 and 48 based on the speed of passing encodermarks can be determined. In one embodiment, dedicated electronics maydetermine velocity and/or acceleration data to minimize loading problemson the user device.

Transducer system also preferably includes actuators 64 to transmitforces to mouse 12 in space, i.e., in two (or more) degrees of freedomof the user object. The bottom housing plate 65 of actuator 64 a isrigidly coupled to ground member 42 (or grounded surface 34) and amoving portion of actuator 64 a (preferably a coil) is integrated intothe base member 44. The actuator 64 a transmits rotational forces tobase member 44 about axis A. The housing 65 of the grounded portion ofactuator 64 b is rigidly coupled to ground member 42 or ground surface34 through the grounded housing of actuator 64 b, and a moving portionof actuator 64 b is integrated into base member 48. Actuator 64 btransmits rotational forces to link member 48 about axis A. Thecombination of these rotational forces about axis A allows forces to betransmitted to mouse 12 in all directions in the planar workspaceprovided by linkage 40 through the rotational interaction of the membersof linkage 40. The integration of the coils into the base members 44 and48 is advantageous and is discussed below. The operation of theelectromagnetic actuators 64 is described in greater detail in U.S. Pat.Nos. 6,100,874 and 6,166,723. In other embodiments, other types ofactuators can be used and is thus not limited to electromagneticactuators, piezoelectric actuators, electroactive polymers, standardspeakers, E-core type actuators, solenoids, eccentric mass motors,linear repetitive motors, and DC motors.

As stated above, in an embodiment the user device is a touch panel, alsoknown as a touch screen, a touch pad, and a touch screen display. Atouch panel includes a touch-sensitive input panel and a display devicecapable of providing a user with a machine or computer interface inwhich the panel is sensitive to the user's touch and displays contentthat responds to the user digit(s) touching the panel. In an embodiment,the touch panel is a planar rectangular pad which can be installed in,on, or near a computer, an automobile, ATM machine, kiosk, mobile phone,Personal Digital Assistant (PDA), media player, etc. In an embodiment,the touch panel fits over a display in which graphical objects (e.g.,icons, menu items, cursors, etc.) are selectable and/or manipulatable bythe user's touch. In an embodiment, the touch panel incorporates hapticcapability in which the touch panel is capable of providing a hapticfeedback or force feedback effect to a user's finger when the usertouches the touch panel. In an embodiment, the touch panel is capable ofoutputting one or more unique haptic sensations for one or moregraphical objects that the user selects via input in the touch panel.For example, the touch panel with haptic capability allows a user tofeel the graphical objects without visually seeing the touch panel. Thisis advantageous in situations where the user may need to visuallyconcentrate on other things going on (e.g., driving a car whileselecting a radio station through the car's stereo).

FIG. 2 illustrates the user device being a touch panel in accordancewith an embodiment. In an embodiment, the user device 70 includes atouch-sensitive panel 72, a display panel 74, and a host computer 76coupled to the display panel 74. In an embodiment, the host computer 76is separate from the display panel 74, as shown in FIG. 2, wherein thehost computer 76 runs a software application program which provides thegraphical objects and graphical environment to the display panel 74 tobe displayed to the user. In an embodiment, the host computer 76 and thedisplay panel 74 are integrated as one device upon which thetouch-sensitive panel 72 disposed. Such an embodiment is applicable in amobile phone, game player, media player, or PDA utilizing a touch panel.

In an embodiment, the touch-sensitive panel 72 is made of a transparentmaterial and is capable of transmitting light from the display panel 74therethrough to the user, so that objects or images displayed by thedisplay panel 74 are able to be easily seen through the touch-sensitivepanel 72. The touch panel employs various types of touch sensingtechnology including, but not limited to, capacitive sensing, resistivesensing, and others known in the art to detect the user's digits atvarious locations on the panel. In an embodiment, a user contacts thetouch panel showing a graphical object thereon to emulate a buttonpress. It is also possible for the user to move his or her finger alongthe panel 72 to access certain features, perform functions and/or selectitems which are displayed. In an embodiment, the touch-sensitive panel72 is further divided into various regions 78 separated by borders 80which may or may not be visible on the touch-sensitive panel 72.

As will be discussed in more detail below, the touch panel user deviceoperates as such in an embodiment where the speed of the user'sfingertip on the touch-sensitive panel correlates to the distance thatthe graphical object is manipulated in the graphical environment. Forexample, if the user moves his or her finger quickly across thetouch-sensitive panel, the graphical object (e.g., a cursor, avatar,etc.) is moved a greater distance in the graphical environment than ifthe user moves the fingertip more slowly.

The user device shown in FIG. 2 further includes one or more actuators82 which apply haptic forces which are transmitted to thetouch-sensitive panel 72 when the actuators 82 are activated. In anembodiment, the actuators 82 excite motion in the plane of thetouch-sensitive panel 72. In an embodiment, actuators 82 excite motionout of the plane of the touch-sensitive panel 72 (e.g., orthogonal tothe plane and/or at an angle to the plane of the touch-sensitive panel72). In an embodiment, all the actuators are energized simultaneously tooutput a cumulative haptic effect to the touch-sensitive panel 72. In anembodiment, one or more actuators are sequentially and/ornon-simultaneously energized to output different haptic effects to thetouch-sensitive panel 72 depending on the location where the usertouches the touch-sensitive panel 72. Using one or more actuatorscoupled to the touch-sensitive panel 72, a variety of haptic sensationscan be output to the user who is contacting the panel. For example,jolts, vibrations (varying or constant amplitude), and textures can beoutput by the actuators. Forces output on the panel 72 can be, at leastin part, based on the location of the finger on the panel; on the stateof a graphical object in the graphical environment; and/or on an eventthat is independent of finger position or graphical object state. Suchindependent forces output on the panel 72 can be “computer-controlled”since a microprocessor (local or remote) from the display device 74controls the magnitude and/or direction of the force output of theactuator(s) using electronic signals. The types of actuators which maybe used with the touch panel user device include piezoelectricactuators, electroactive polymers, voice coil actuators, standardspeakers, E-core type actuators, solenoids, eccentric mass motors,linear repetitive motors, DC motors, and moving magnet actuators.

The display 74 can be any type of display such as a cathode ray tube(CRT), liquid crystal display (LCD), plasma display, flat-panel display,or could even be a static illustration. In an embodiment, the displaypanel 74 includes a local controller or microprocessor, local memory,and data bus for data processing. In an embodiment, a bus 84 is used tofacilitate data or signal transfer between the display panel 74 and thetouch-sensitive panel 72. In an embodiment, conductive integratedconnections may be implemented on the display panel 74 and thetouch-sensitive panel 72, such that data and signal transfer can occurbetween the two without the need for wires and upon the touch-sensitivepanel 72 being mounted to the display panel 74. In an embodiment, thetouch sensitive panel 72 and computer may communicate with one anotherwirelessly using a wireless media transmitting through personal areanetwork (“PAN”) and/or wireless local area network (“WLAN”).

In an embodiment, the host computer 76 is a digital processing system,workstation, server, mini-computers, or mainframe computers. Forexample, host computer 76 can operate under the Windows.™. or MS-DOSoperating system in conformance with a personal computer standard. Thehost computer 76 includes a host microprocessor, random access memory(RAM), read only memory (ROM), input/output (I/O) circuitry, and othercomponents such as wireless communication circuitry and networkcommunication circuitry. In an embodiment, the host computer 76implements one or more application programs with which a user interactsvia the touch panel 70 or other user device utilizing the force feedbackfunctionality afforded to the user device.

For example, the host application programs can include a simulation,video game, Web page or browser that implements HTML or VRMLinstructions, word processor, drawing program, spreadsheet, scientificanalysis program, Point of Sale (POS) program, or other applicationprogram that utilizes input of the touch panel user device 70 andoutputs force feedback commands to the touch panel user device 70 to beoutput to the user as haptic effects. In another example, the touchpanel user device 70 can display images of a GUI supplied by the hostcomputer 76. Graphical objects such as a first person point of view canbe displayed via the display panel 74, as in a virtual reality game.Additionally or alternative, graphical objects representing athird-person perspective of objects, backgrounds, etc. can be displayedvia the user device.

FIG. 3 is a block diagram illustrating the electronic portion ofinterface 14 and host computer 18 in accordance with an embodiment. Thesystem 10 includes a host computer 18, electronic interface 26,mechanical portion 24, and the user device. Electronic interface 26 andthe user device are collectively considered the “force feedbackinterface device” 11 that is coupled to the host computer. In anembodiment, “force feedback interface device” includes the mechanicalportion 24, depending on whether the user device has mechanicalcomponents. For example, the force feedback interface device for a userdevice which is a mouse would include a mechanical portion 24 whereas auser device which is a touchscreen or touch panel would not necessarilyinclude a mechanical portion 24.

Host computer system 18 commonly includes a host microprocessor 108,random access memory (RAM) 110, read-only memory (ROM) 112, input/output(I/O) electronics 114, a clock 116, a display device 20, and an audiooutput device 118. Host microprocessor 108 can include a variety ofavailable microprocessors from Intel, AMD, Motorola, or othermanufacturers. Microprocessor 108 can be single microprocessor chip, orcan include multiple primary and/or co-processors. Microprocessor 108preferably retrieves and stores instructions and other necessary datafrom RAM 110 and ROM 112 as is well-known to those skilled in the art.In the described embodiment, host computer system 18 can receive sensordata or a sensor signal via a bus 120 from sensors of system 10.Microprocessor 108 can receive data from bus 120 using I/O electronics114, and can use I/O electronics to control other peripheral devices.Host computer system 18 can also output commands to interface device viabus 120 to cause force feedback for the interface system 10.

Clock 116 is a standard clock crystal or equivalent component used byhost computer 18 to provide timing to electrical signals used by hostmicroprocessor 108 and other components of the computer system 18.Display device 20 has been described above. Audio output device 118,such as speakers, can be coupled to host microprocessor 108 viaamplifiers, filters, and other circuitry well-known to those skilled inthe art. Host processor 108 outputs signals to speakers 118 to providesound output to the user when an “audio event” occurs during theimplementation of the host application program. Other types ofperipherals can also be coupled to host processor 108, such as storagedevices (hard disk drive, CD ROM drive, floppy disk drive, integrated orremote server, etc.), printers, and other input and output devices.

In an embodiment, the electronic interface 26 is coupled to hostcomputer system 18 by a bi-directional bus 120. The bi-directional bussends signals in either direction between host computer system 18 andthe interface device. Bus 120 can be a serial interface bus providingdata according to a serial communication protocol, a parallel bus usinga parallel protocol, or other types of buses. An interface port of hostcomputer system 18, such as an RS232 serial interface port, connects bus120 to host computer system 18. In another embodiment, an additional buscan be included to communicate between host computer system 18 andinterface device 11. In an embodiment, a USB standard connector providesa relatively high speed serial interface that can provide force feedbacksignals with a high degree of realism. USB can also source power todrive actuators 64 and other devices. In addition, the USB standardincludes timing data that is encoded along with differential data.Alternatively, Firewire (also called 1392) can be used as bus 120; or,the bus can be between an interface card in the host computer 18, wherethe interface card holds components of device 11 such as microprocessor130.

In an embodiment, the electronic interface 26 includes a localmicroprocessor 130, local clock 132, local memory 134, sensor interface136, and actuator interface 138. Interface 26 may also includeadditional electronic components for communicating via standardprotocols on bus 120. In various embodiments, electronic interface 26can be included in mechanical portion 24, in host computer 18, or in itsown separate housing. Different components of interface 26 can beincluded in portion 24 or host computer 18 if desired.

Local microprocessor 130 is preferably coupled to bus 120 and may beclosely linked to allow quick communication with other components of theinterface device. Processor 130 is considered “local” to interfacedevice 11, where “local” herein refers to processor 130 being a separatemicroprocessor from any processors 108 in host computer 18. “Local” alsorefers to a processor 130 being dedicated to force feedback and sensorI/O of the interface system 10, and being closely coupled to sensors andactuators of the mechanical portion 24, such as within the housing of,or in a housing coupled closely to, portion 24. Microprocessor 130 canbe provided with software instructions to wait for commands or requestsfrom computer host 18, parse/decode the command or request, andhandle/control input and output signals according to the command orrequest. In addition, processor 130 preferably operates independently ofhost computer 18 by reading sensor signals and calculating appropriateforces from those sensor signals, time signals, and force processesselected in accordance with a host command, and output appropriatecontrol signals to the actuators. A suitable microprocessor for use aslocal microprocessor 130 includes the 8X930AX by Intel; or alternativelythe MC68HC711E9 by Motorola or the PIC16C74 by Microchip, for example.Microprocessor 130 can include one microprocessor chip, or multipleprocessors and/or co-processor chips. In an embodiment, microprocessor130 can include digital signal processor (DSP) functionality.

For example, in one host-controlled embodiment that utilizesmicroprocessor 130, host computer 18 can provide low-level forcecommands over bus 120, which microprocessor 130 directly transmits tothe actuator(s). In a different local control embodiment, host computersystem 18 provides high level supervisory commands to microprocessor 130over bus 120, and microprocessor 130 manages low level force controlloops to sensors and actuators in accordance with the high levelcommands and independently of the host computer 18. In the local controlembodiment, the microprocessor 130 can process inputted sensor signalsto determine appropriate output actuator signals by following theinstructions of a “force process” that may be stored in local memory andincludes calculation instructions, formulas, force magnitudes, or otherdata. The force process can command distinct force sensations, such asvibrations, textures, jolts, or even simulated interactions betweendisplayed objects. An “enclosure” host command can also be provided,which causes the microprocessor to define a box-like enclosure in agraphical environment, where the enclosure has sides characterized bywall and texture forces. The host can send the local processor a spatiallayout of objects in the graphical environment so that themicroprocessor has a mapping of locations of graphical objects likeenclosures and can determine interactions with the cursor locally.

Sensor signals used by microprocessor 130 are also reported to hostcomputer system 18, which updates a host application program and outputsforce control signals as appropriate. For example, if the user operatesthe user device, the computer system 18 receives position and/or othersignals indicating this input and can move a displayed graphical objectin response. In an alternate embodiment, no local microprocessor 130 isincluded in interface system 10, and host computer 18 directly controlsand processes all signals to and from the interface 26 and user device.

In an embodiment, a local clock 132 is coupled to the microprocessor 130to provide timing data, similar to system clock 116 of host computer 18;the timing data might be required, for example, to compute forces outputby actuators 64 (e.g., forces dependent on calculated velocities orother time dependent factors). In an embodiment using the USBcommunication interface, timing data for microprocessor 130 can beretrieved from the USB interface. Local memory 134, such as RAM and/orROM, is preferably coupled to microprocessor 130 in interface 26 tostore instructions for microprocessor 130 and store temporary and otherdata. Microprocessor 130 may also store calibration parameters in alocal memory 134 such as an EEPROM. Memory 134 may be used to store thestate of the force feedback device, including a reference position,current control mode or configuration, etc.

In an embodiment, the sensor interface 136 is included in electronicinterface 26 to convert sensor signals to signals that can beinterpreted by the microprocessor 130 and/or host computer system 18.For example, sensor interface 136 can receive signals from a digitalsensor such as an encoder and convert the signals into a digital binarynumber representing input information. An analog to digital converter(ADC) in sensor interface 136 can convert a received analog signal to adigital signal for microprocessor 130 and/or host computer 18. Suchcircuits, or equivalent circuits, are well-known to those skilled in theart. Alternately, microprocessor 130 or host 18 can perform theseinterface functions.

In an embodiment, the actuator interface 138 is connected between theactuators 64 and microprocessor 130. Interface 138 converts signals frommicroprocessor 130 into signals appropriate to drive the actuators.Interface 138 can include power amplifiers, switches, digital to analogcontrollers (DACs), and other components. Such interfaces are well-knownto those skilled in the art. In alternate embodiments, interface 138circuitry can be provided within microprocessor 130 or in the actuators.

In the described embodiment, power is supplied to the actuators 64 andany other components (as required) by the USB. Since the electromagneticactuators of the described embodiment have a limited physical range andneed only output, for example, about 3 ounces of force to createrealistic force sensations on the user, very little power is needed. Forexample, one way to draw additional power from the USB is to configuredevice 11 to appear as more than one peripheral to host computer 18; forexample, each provided degree of freedom of mouse 12 can be configuredas a different peripheral and receive its own allocation of power.Alternatively, power from the USB can be stored and regulated by device11 and thus used when needed to drive actuators 64. For example, powercan be stored over time and then immediately dissipated to provide ajolt force to the user object 12. A battery or a capacitor circuit, forexample, can store energy and discharge or dissipate the energy whenpower is required by the system and/or when enough power has beenstored. In an embodiment, a power supply 140 is coupled to the actuatorinterface 138 and/or actuators 64 to provide electrical power. The powerstorage embodiment described above, using a battery or capacitorcircuit, can also be used in non-USB embodiments to allow a smallerpower supply 140 to be used.

In an embodiment in which the user device utilizes a mechanical portion23, the mechanical portion 24 is coupled to electronic portion andincludes sensors 62, actuators 64, and linkage 40. Sensors 62 sense theposition, motion, and/or other characteristics of user device 12 alongone or more degrees of freedom and provide signals to microprocessor 130including information representative of those characteristics.Typically, a sensor 62 is provided for each degree of freedom alongwhich the user device can be moved, or, a single compound sensor can beused for multiple degrees of freedom. Example of sensors suitable forembodiments described herein are rotary optical encoders, as describedabove, linear optical encoders, analog sensors such as potentiometers,or non-contact sensors, such as Hall effect magnetic sensors or anoptical sensor such as a lateral effect photo diode having anemitter/detector pair. In an embodiment, velocity sensors (e.g.,tachometers) and/or acceleration sensors (e.g., accelerometers) areused. Furthermore, either relative or absolute sensors can be employed.

Actuators 64 transmit forces to the user device in one or moredirections along one or more degrees of freedom in response to signalsoutput by microprocessor 130 and/or host computer 18, whereby they are“computer controlled.” In an embodiment, an actuator 64 is provided foreach degree of freedom along which forces are desired to be transmitted.Actuators 64 can be the electromagnetic actuators described above, orcan be other active actuators, such as linear current control motors,stepper motors, pneumatic/hydraulic active actuators, a torquer (motorwith limited angular range); or passive actuators such as magneticparticle brakes, friction brakes, or pneumatic/hydraulic passiveactuators, passive damper elements, an electrorheological fluidactuator, a magnetorheological fluid actuator. In addition, in voicecoil embodiments as well as multiple wire coils can be provided, whereinsome of the coils can be used to provide back EMF and damping forces. Inan embodiment, all or some of sensors 62 and actuators 64 can beincluded together as a sensor/actuator pair transducer. It should benoted that the actuator may be applied to any type of user devicedescribed above even though the mouse 12 is shown in FIG. 1 a. In theembodiment in which the user device utilizes the mechanical linkageshown or a variation thereof, the user device may incorporate a puck,knob, joystick, stylus, or other device as described above. These otherinput devices 141 can optionally be included in system 10 and send inputsignals to microprocessor 130 and/or host computer 18. Such inputdevices can include buttons, such as buttons 15 on mouse 12, used tosupplement the input from the user to a GUI, game, simulation, etc.Also, dials, switches, voice recognition hardware (with softwareimplemented by host 18), or other input mechanisms can be used.

In an embodiment, safety or “deadman” switch 150 is preferably includedin interface device to provide a mechanism to allow a user to deactivateactuators 64 for safety reasons. In the preferred embodiment, the usermust continually close safety switch 150 during manipulation of mouse 12to activate the actuators 64. If, at any time, the safety switch isdeactivated (opened) the actuators are deactivated while the safetyswitch is open. For example, one embodiment of safety switch is acapacitive contact sensor that detects when the user is contacting mouse12. Alternatively, a mechanical switch, electrostatic contact switch, oroptical switch located on mouse 12 or on a convenient surface of ahousing 21 can be used. A z-axis force sensor, piezoelectric sensors,force sensitive resistors, or strain gauges can also be used. The stateof the safety switch can be sent to the microprocessor 130 or host 18.In an embodiment, mouse 12 includes a hand weight safety switch. Thesafety switch preferably deactivates any generated forces on the mousewhen the mouse is not in use and/or when the user desires to deactivateoutput forces.

In an embodiment, multiple mechanical apparatuses 102 and/or electronicinterfaces 100 can be coupled to a single host computer system 18through bus 120 (or multiple buses 120) so that multiple users cansimultaneously interface with the host application program (in amulti-player game or simulation, for example). In addition, multipleplayers can interact in the host application program with multipleinterface systems 10 using networked host computers 18, as is well-knownto those skilled in the art. Also, the interface device can be coupledto multiple host computers; for example, a local host computer candisplay images based on data received from a remote host computercoupled to the local host through a network.

Architecture of Host Computer

The host computer 18 interacts with interface device 11 using a numberof different levels of controllers. These controllers are preferablyimplemented in software, e.g., program instructions or code; however,all or part of the controllers may also be implemented in hardware,wherein the conversion of functionality of the software to hardware iswell-known to those skilled in the art.

The architecture described herein is oriented towards providing forcefeedback functionality to a host computer connected to a force feedbackinterface device, where multiple applications may be runningsimultaneously on the host computer. Each application program may haveits own set of force sensations to output to the device; since thedevice cannot implement all force sensations at any one time due to costand hardware constraints, the forces commanded by each application mustbe organized by the architecture to take these limitations into account.

FIG. 4 is a block diagram of an architecture for the host computer tocommunicate with and control a force feedback user device 11 withmultiple application programs running on the host computer. Applicationprograms 202 and 204 are concurrently running on the host computer 18.In the most typical implementation, one of the application programs isactively running in an operating system as the “active” applicationprogram that displays one or more active windows in the GUI (also knownas the application program that is “in focus” or which has “keyboardfocus”). The active window is typically the topmost displayed window inwhich input is provided by the user using the mouse-controlled cursor, akeyboard, or other peripheral. The other applications are “inactive” inthat they are not receiving input from the user (although they may havea window displayed in the GUI which can be updated on the screen). Theinactive applications may also receive input or send output. Forexample, the Windows.™. operating system from Microsoft Corp. provides amultitasking or pseudo-multitasking environment in which programs runsimultaneously; other operating systems such as Unix also offermultitasking. For example, a word processor may be the activeapplication to receive input from the keyboard and display inputcharacters in a displayed active window on display screen 20, while aninactive communication program may be receiving data over a network andsaving the data on a storage device, or sending data out to be printed.Or, an inactive drawing program may be displaying a graphical image in awindow that is not currently active, while the user inputs text into theactive word processor window. When the user moves the cursor over aninactive window and provides a command gesture such as clicking a buttonon a mouse, the inactive window becomes the active window and the formeractive window becomes inactive. The active application is also known asthe “foreground” application, in the sense that its force sensations arebeing implemented by the force feedback device, as described below.

A master application 206 also is running on host computer 18 and is alsoknown as the “background” force feedback application. In an embodiment,this application is a general purpose program that always runsinactively in the operating system and whose set of commanded forces arealways available to be output and controlled at the user device.

In an example embodiment, the master application is a “desktop” controlpanel as illustrated in FIG. 5. In an embodiment, a “mouse properties”dialog box 240 can be displayed which allows the user to specify forcesensations 244 that are assigned to specified object types 242 in thedisplayed graphical environment, e.g., a graphical user interface (GUI).It should be noted that although the dialog box 240 is shown to apply tomouse properties, dialog box 240 is applicable to other user devicessuch as a touchpad, touchscreen, joystick, etc. For example, theassigned force sensation 244 can be a vibration force having a specifiedfrequency, magnitude, duration, etc.; a single “pop” or jolt at aspecified time and duration; a “tick” that is a small jolt that isoutput based on manipulation of the user object or graphical object atregular intervals during an activity such as scrolling; a damping forcecondition that causes a resistance force to the user device depending onthe velocity of the user device in its degrees of freedom or thegraphical object in the host frame on screen 20; or a spring thatresists motion of the user object based on distance to an origin of thespring.

In an embodiment, the force sensations specified by the user for theuser device in the dialog box 240 will be output by the user device bydefault, unless a different foreground application program deactivatesthe force sensations or replaces a force sensation with its own. Forexample, the user can specify that menu objects 242 a in a GUI have asnap force 244 a associated with them by moving the slider 246 to theright, wherein the slider scale specifies the magnitude of the snapforce. Thus, when the user selects a menu item in a menu during normaloperation, the user will feel a haptic effect. In an embodiment, thehaptic effect is a snap force, i.e., a force that draws the user objecttoward the center of the menu item to allow easier menu selection. In anembodiment, the haptic effect is a vibration that is felt upon theuser's finger. This haptic effect is applied to all menus of all runningapplication programs in the GUI since it is a background force effect.For example, if the foreground application program has its own forcesensations which define that application's menus to have a jolt insteadof a snap, then the jolt will be superimposed on the snap unless theactive application program deactivates the background force(s). Ingeneral, a particular active application program will typically commandforces to objects in its own windows, e.g., that application's ownmenus, scrollbars, icons, window borders, etc.

In an embodiment, a user can specify multiple background forcesensations 244 for each user interface object 242. This allows themultiple force sensations to be superimposed on each other, unless theapplication program overrides one or more of the superimposed forces.Thus, a user can assign a damper force sensation and a “ticks” forcesensation to scrollbars in box 240, and all scrollbars will have theseforces associated with them unless overridden by the foregroundapplication program.

Other controls in box 240 may include a device gain 248 to set thepercentage of maximum magnitude for all the forces for items 242; andselections 249 to allow force feedback functionality on the hostcomputer. Advanced button 250, when selected, displays an additionalwindow of options with which the user can customize the force sensations244. For example, a user can specify positive or negative damping (wherenegative damping feels like moving on ice) or unidirectional orbidirectional dampers. The user can specify the spacing or orientationof snap points on a grid, the envelope parameters for a vibration, orthe parameters for a snap force, e.g., the snap force is defined as anenclosure and the user can customize the enclosure to turn off forces onone wall, turn on inner walls, etc. Other application programs can alsobe used as the background or master application program 206.

Referring back to FIG. 4, application programs 202, 204, and 206communicate with a force feedback Application Programming Interface(“API”) 208 which is resident in the host computer's memory and whichallows a given application program to communicate with lower leveldrivers and other force feedback functions. For example, in theWindows.™. operating system, APIs are commonly used to allow a developerof an application program to call functions at a high level for use withthe application program, and not worry about the low level details ofactually implementing the function.

The API shown in the embodiment in FIG. 4 includes a set of software“objects” that can be called to command a force feedback interfacedevice 11. Objects are a set of instructions and/or data which can becalled by a pointer and/or an identifier and parameters to implement aspecified function. For example, three types of objects are provided inone preferred API implementation: interface objects, device objects, andeffect objects. Each of these objects makes use of one or more forcefeedback device drivers which are implemented as objects in the API 208.

Interface objects represent the API at the highest level. An applicationprogram (which is referred to as a “client” of the API) can create aninterface object to access lower level objects and code that implementforce feedback device functionality. For example, the interface objectallows an application program to enumerate and receive information aboutindividual devices and create lower level objects for each device usedby the application program.

Device objects each represent a physical force feedback device (or othercompatible device or peripheral) in communication with the host computer18. For example, the force feedback mouse device described in FIG. 1 awould be represented as a single device object, whereas a touch panelwould be represented as another single device object. The device objectsaccess the set of force feedback routines to receive information aboutan associated physical device, set the properties of the physicaldevice, register event handlers (if events are implemented on the host),and to create effect objects.

Effect objects each represent a force feedback effect defined by theapplication program to be output one or more times on a force feedbackuser device. The effect objects access the set of force feedbackroutines to download force effects to the force feedback user device,start and stop the output of effects by the force feedback user device,and modify the parameters of the effects. Event objects can also becreated to define events, as described in greater detail below.

A force “effect,” as referred to herein, is a single defined force orseries of forces that may be commanded within the API or by the userdevice itself The effect typically has a name to identify it and one ormore parameters to modify or customize the force as necessary. Forexample, several types of force effects have been defined, includingvibrations, enclosures, grids, textures, walls, dampers, snapsensations, vibrations, circles, ellipses, etc. For example, a vibrationforce effect may have parameters of duration, frequency, magnitude, anddirection. The force sensations 244 defined in dialog box 244 can beforce effects; however, more generally, force sensations can be made upof one or more force effects. Force effects, in turn, are made up of oneor more basic force prototypes, such as springs, dampers, and vectorforces.

In an embodiment, an application program interacts with the API 208 byfirst receiving a pointer to a resident force feedback interface; forexample, one type of interface includes procedures provided in theComponent Object Model (COM) interface of Microsoft Windows.™., anobject-oriented interface. Using the force feedback interface, theapplication program enumerates the force feedback devices on thecomputer system 18. The application program selects a desired one of theforce feedback devices and creates a device object associated with thatdevice. Using the force feedback interface, the application thenacquires the device, sets the device up with setup and defaultparameters, and creates effect objects and event objects at timesdesignated by the developer of the application. At appropriate times,the application program will command the start, stop, or pause of theplayback of force effects by accessing the appropriate interfaceinstructions associated with the desired effect. The API indicates tothe context driver 210 to create a “context” (i.e., “effect set”) for anapplication program, and sends effects and events to be associated withthat context. A “context” is a group or set of effects and events thatare associated with a particular application program.

In an embodiment, the context driver 210 receives effect set 222 anddevice manipulation data 223 from the API 208. The context driver 210 isprovided at a level below the API to organize contexts for applications202 and 204 running on the host computer. In an embodiment, the effectsand events in a context are not known to the application program itself;rather, the context driver 210 creates the context internally for anapplication. Thus, an application commands that a particular forcesensation be output in response to different interactions oroccurrences, e.g., an interaction of a graphical object with anotherobject or region in the graphical environment, or the output of a forcebased on other criteria (time, received data, random event, etc.). TheAPI sends the commanded effect(s) to the context driver 210, and thecontext driver stores the effects in the created context for thatapplication program.

Since each application may have a completely different set of forceeffects to output, each context should be associated with its particularapplication program. In an embodiment, there are foreground contexts andbackground or primary contexts. A foreground context is associated withthe application program 202 or 204 that is active in the operatingsystem. A background context includes the effects and events for themaster application program 206, e.g., the effects and events whichimplement those force sensations selected by the user in the dialog box240 of FIG. 5 to be associated with particular GUI objects. In anembodiment, a “global context” may be provided, which includes commoneffects almost always used by application programs (e.g., a pop force)and which can automatically be downloaded to the force feedback userdevice. Effects in the global context need not be stored in individualcontexts for particular application programs and need not be sent to theforce feedback device, thus saving memory on the device.

When an application program is first executed by the host computer andloaded into memory, if the application is able to command a forcefeedback user device, the application will query for the API 208. Oncecommunication is established, the API will contact the context driver210 to create an entry location for a context set for the initiatedapplication program.

In an embodiment, the context driver 210 receives individual effects andevents as they are created by the application program using the API 208and stores the effects and events in a context list 212, storing eachcontext in a different storage location in the host's memory or on someother type of storage device. In an embodiment, an active or inactiveapplication program can create a context and have it stored, but onlythe active application's context will be sent to the force feedback userdevice. The context driver 210, in an embodiment, can examine anidentifier in a created effect or event to determine which applicationprogram is associated with it and thus store the effect or event in theproper memory location. The context driver sets pointers to the contextsand to the effects and events in the contexts to access them. The effectcan be created initially, before any forces are commanded, or the effectcan be created and then immediately commanded to be output to the forcefeedback device. Each context also preferably includes an entry into adevice state structure. This entry governs the “gain” or force level forall effects in the context. For example, all the forces in the contextcan be cut to 50% of full magnitude by storing a value of 50 in thedevice state structure. One of the contexts stored in list 214 is aprimary context 216, which is the list of effects and events used by themaster application 206 or “background” application.

After the context driver has stored the contexts in list 212, anapplication program may send a command to the API to output or “start” aparticular force sensation. In an embodiment, the API checks whether theapplication program is active or in the background (primary); if not,the API ignores the command. In an embodiment, the commands from aninactive foreground application can be stored and then sent to thedevice when the application becomes active. If the application is activeor background, the API sends the start information to the context driver210 indicating the application program that commanded the force and theparticular force effects to be commanded. The context driver 210 thenassociates the commanding application program with a context 214 in list212 and sends the effects from the context to the force feedback device(if not previously sent). For example, if a context for a particularapplication program includes a spring effect, a damper effect, and avibration effect, and the application program commands the vibration tobe output, then the context driver selects the vibration effects to beoutput to the device. The data describing this effect is then output bythe context driver 210. Similarly, the application program can send acommand to stop particular force effects, to pause the effects, to getthe status information of an effect, or to destroy an effect.

A context is preferably only allowed to exert forces with the forcefeedback device when that context is active, i.e., is associated with anactive application program or the background application. In anembodiment, only one foreground context can be active at any given time.In another embodiment, more than one foreground context can be active atany given time. Any number of background contexts may be simultaneouslyactive; however, there may be a device limit on the number of backgroundcontexts that may be created. For example, the user device may onlyallow one background context to be created at any one time. Thus, if aninactive (not in focus) foreground application program commands a forceto be output, the API will ignore the command after determining that thecommanding application is not active (or, the command will only be sentto the device when the application becomes active).

In an embodiment, if the active application program becomes inactive(e.g., loses foreground or is removed from the host's memory) and adifferent application becomes active, then the API indicates this to thecontext driver 210 which then deactivates the context associated withthat application program and loads the effects from the new activecontext to the force feedback device 11. Likewise, when the originalapplication program again becomes active, the API instructs the contextdriver 210 to activate the associated context and load the appropriateeffects to the force feedback device.

Device manipulation data 223 is also received by the context driver 210.Data 223 received from the user device is used to set a global devicestate on the force feedback user device, as described below. Thisinformation is passed unmodified to the translation layer 218. Thetranslation layer 218 manages the sending of force feedback effects tothe user device, receives information sent from the user device to thehost computer, and maintains a virtual representation of the memory ofuser device. The translation layer 218, in an embodiment, receives anindividual effect 219 for the active (or background) application programand device manipulation data 223 sent by the context driver 210. Thetranslation layer 218 receives the effect from a context list 220 ofindividual effects 222 (list 220 represents a context 214). A differentcontext list 220 is provided in each context 214 of list 212. Eacheffect 222 in list 220 is a force that is to be output on the mouse 12by the force feedback device 11. Then the effects are to be sent to thedevice 11 when commanded by the application, they are selected andcopies are output to the device. In an embodiment, each effect issubstantially simultaneously output by the translation layer in theorder of receiving the effects. Each effect stored in list 220, asexamined by the translation layer is available on force feedback device11, such that the local microprocessor should recognize the effect andbe able to output the effect either immediately or when conditionsdictate. The size of the list 220 should be the same or smaller than theavailable memory for such a list in the force feedback device so thatthe translation layer 218 knows when the memory of the force feedbackdevice is full. If the memory full, the translation layer 218 can delaythe output of additional effects until enough memory space is available.

When an active application becomes inactive, the translation layer 218is instructed by the context driver 210 to “unload” the effects of thecontext of the previous active application from the force feedback userdevice. The context driver 210 receives the effects from the activeapplication and loads those effects to the force feedback user device(the effects for the background (primary) application are preferablynever unloaded).

The translation layer 218 also handles events, in an embodiment. Forexample, when a screen position is received from the user device, thetranslation layer 218 can check whether any of the conditions/triggersof the active events are met. If so, a message is sent which reaches theassociated active or background application program, as described ingreater detail below. In an embodiment, the microprocessor 130 on device11 can check for events and send event notifications through thetranslation layer 218 up to the application program. However, this canbe undesirable in some embodiments since the event notification isprovided at a much slower rate than the force control loop of themicroprocessor 130.

In an embodiment, the translation layer also can store a device state224 in memory. Device manipulation data 223 from the active applicationdetermines the device state, which is the state that the activeapplication program wishes to impose on the device, if any. For example,an overall condition can be stored, such as an enable or disable for allforces, so that if all forces are disabled, no forces will be output bythe device. An overall gain can also be set to limit all output forcemagnitudes to a desired level or percentage of maximum output.

In an embodiment, the translation layer outputs device messages 225 tothe device 11 by way of the next layer. Messages may include forceeffects to be output and/or any other information such as deviceidentification numbers or instructions from the context driver for aneffect (start, stop, pause, reset, etc.) The translation layer outputsmessages 225 in a form the user device is able to interpret.

In an embodiment, the device communication driver 226 communicatesdirectly with the force feedback user device. Driver 226 receives thedevice messages 225 from translation layer 218 and directly transmitsthe messages to the force feedback device over bus 120, e.g., a USB, inan embodiment. The driver is implemented, in an embodiment, as astandard driver to communicate over such a serial port of host computer18. Other types of drivers and communication interfaces are contemplatedand are not limited to the types of drivers described above.

Host Command Structures

The host computer 18 and API 208 handle a variety of processescontrolling force feedback user device. Different effects and events aredescribed below.

Effects

Effects are standardized forces that are determined according to apredefined characterization, which may include force algorithms, storedforce magnitudes, functions of space and/or time, a history of storedmotion values of the mouse, or other steps or parameters. An effect maybe based on time and be independent of the manipulation input by theuser, or may be spatially determined based on position, velocity,acceleration, or any combination of these. In an embodiment, the userdevice incorporates a number of effects standardized in its memory whichit can implement if the effects are within the active or backgroundcontext. When the device is determining output forces based on effects,the microprocessor 130 checks if the effect is active, calculates theraw contribution to the output force from the effect, applies anenvelope scaling, and sums the scaled contribution to the total sum offorces contributed to by all effects currently being output. Indetermining the total sum, the user device, in an embodiment, combinesall constant forces (e.g., conditions and time-based forces) and limitsthe constant force sum to a predetermined magnitude. The user devicethen combines all dynamic forces and limits the dynamic force sum to apredetermined magnitude, in an embodiment. The two sums are then addedtogether and the total force sum is output by the actuators of the userdevice. Alternatively, all forces can be treated the same and summedtogether.

FIGS. 6 a and 6 b illustrate one type of force effect provided by thehost computer in a graphical environment known as an “enclosure.” In anembodiment, an enclosure is a rectangular object provided in a GUI whichmay be assigned with different forces to be output by the user devicefor each of its portions or sides. The enclosure is a type of forcefeedback object that can be used for a variety of different graphicalobjects, including buttons, windows, sliders, scrollbars, menus, linkson a web page, or grid points in a drawing program.

FIG. 6 a illustrates the use of an enclosure for a window in a GUI. Thewindow 260 is displayed on the computer screen of the host computer, inan embodiment. In an embodiment of the window is displayed on the userdevice itself (e.g., touch screen, touch panel). The enclosure has eightwalls 262 associated with it invisible to the visual perception of theuser, where four walls are provided as an inner rectangle 264 and fourwalls are provided as an outer rectangle 266. Each wall 262 can beseparately defined as either providing a force pulling the usercontrolled graphical object toward the border of window 260 (attractiveforce), or pushing the user controlled graphical object away from theborder of window 260 (resistive force). The outer rectangle 266 wallsaffect the user controlled graphical object when the user object isoutside the window 260 between the rectangle 266 and the window 260border. In addition, the inner rectangle 264 walls affect the usercontrolled graphical object when it is inside the window 260 between therectangle 264 and the window border. In an embodiment, a window 260 isassociated with a force enclosure by defining both outer rectangle 266and inner rectangle 264 to have an attractive force, so that the userobject and cursor tend to be attracted to the window border when withinthe appropriate range of the window. This, for example, allows the userto acquire the border and adjust the size of the window easily. In anembodiment, the outer rectangle 266 can be defined as a resistive force(such as a spring) that requires the user to overcome the resistance toallow the user object entry into the window (and the rectangle 266 canalso be defined as a click surface to cause a function to be appliedwhen gaining entry). When the user controlled graphical object gainsentry to an enclosure, an “event” occurs and is indicated to theapplication program, as explained above. In an embodiment, enclosurescan be similarly defined for circles, ellipses, triangles, or othershapes, both regular and irregular.

In an embodiment, the enclosure object has a number of other parametersthat further define the enclosure. For example, parameters may bespecified separately for each wall to define such characteristics as themagnitude of the resistive or attractive forces, which of the eightwalls are assigned forces, which walls provide clipping, the saturationlevel of the force applied in association with a wall, and the interiorforce, which can be specified as any force effect such as a damping,texture, or vibration force, that is applied while the graphical objectis inside the enclosure.

FIG. 6 b illustrates another embodiment of an enclosure. A button 270 isshown in FIG. 6 b which can be a graphical button, a menu item, or anicon in a GUI. The outer rectangle 266 shown in dashed lines in FIG. 6 bhas been “turned off” so that no forces are associated with it. Theinner rectangle 264 has been defined as a very small rectangle at thecenter of the button 270. Resistive forces are associated with the innerrectangular walls such that, once the user object has entered theenclosure, a force is applied and biases the user device toward thecenter of the button 270. This allows the user to target or “acquire”the button more easily with the cursor and reduces overshooting thebutton by accident by moving the cursor too far past the button.

Enclosures are quite useful for adding forces to web links on a web pageviewed in a web browser, such as Netscape Communicator by NetscapeCommunications, Firefox, or Internet Explorer by Microsoft. The innerwall forces of FIG. 6 b can be used on a web link displayed on a webpage to draw the graphical object onto a desired link, thus allowing auser to more easily determine which images are links on the page. Othertypes of forces can also serve this function. Also, an “insidecondition” such as a texture can be provided for the web link to causesome links to feel rough, some smooth, some bumpy, etc.

An example of an enclosure effect structure is provided below:

ENCLOSURE typedef struct { DWORD dwSize; RECT rectOutside; DWORDdwXWidth; DWORD dwYWidth; DWORD dwXStiffness; DWORD dwYStiffness; DWORDdwStiffnessMask; DWORD dwClippingMask; DWORD dwXSaturation; DWORDdwYSaturation; LP_CONDITION pInsideCondition } ENCLOSURE;

-   dwSize: Size of the structure in bytes.-   rectOutside: Outside borders of the enclosure, in screen    coordinates. When the mouse position is outside of rectangle defined    by rectOutside, there are no forces exerted on the device from this    enclosure.-   dwXWidth, dwYWidth: Width W of the region between a wall and the    object border. In one implementation, the width of the inner wall    and the outer wall regions is the same. Alternatively, they can be    defined separately. This parameter can alternatively be specified as    a deadband.-   dwXStiffness: Stiffness of the left and right walls. Defines the    amount of horizontal resistance (force) felt by the user when    attempting to break through a vertical wall. This value amounts to    the negative and positive coefficient of horizontal spring    conditions that are applied when the mouse is in the left and right    walls, respectively.-   dwYStiffness: Stiffness of the top and bottom walls. Defines the    amount of vertical resistance (force) felt by the user when    attempting to break through a horizontal wall.-   dwStiffnessMask: Mask specifying when resistance should be applied.    Can be one or more of the following values:    -   NONE: No resistance is felt on any wall.    -   INBOUNDTOPWALL, INBOUNDLEFTWALL, etc.: Resistance is felt when        entering the enclosure through the top, bottom, left, or right        wall, as specified.    -   OUTBOUNDTOPWALL, OUTBOUNDLEFTWALL, etc.: Resistance is felt when        exiting the enclosure through the top, bottom, left, or right        wall, as specified.    -   INBOUNDANYWALL, OUTBOUNDANYWALL: Resistance is felt when        entering or exiting, respectively, the enclosure through any        wall.    -   ANYWALL: Resistance is felt when entering or exiting the        enclosure through any wall.-   dwClippingMask: Mask specifying when mouse cursor clipping should be    applied. Can be one or more of the following values:    -   NONE: Clipping is not applied to any wall    -   INBOUNDTOPWALL, INBOUNDLEFTWALL, etc.: Clipping is applied when        entering the enclosure through the top, bottom, left, or right        wall, respectively.    -   OUTBOUNDTOPWALL, OUTBOUNDLEFTWALL, etc.: Clipping is applied        when exiting the enclosure through the top, bottom, left, or        right wall, respectively.    -   INBOUNDANYWALL, OUTBOUNDANYWALL: Clipping is applied when        entering or exiting, respectively, the enclosure through any        wall    -   ANYWALL: Clipping is applied when entering or exiting the        enclosure through any wall.-   dwXSaturation, dwYSaturation: Saturation level of spring effect felt    by the user when attempting to break through a vertical or    horizontal wall, respectively.-   pInsideCondition: Points to a condition that is to be in effect when    the user mouse is in the deadband area. For example, can specify a    spring condition, damper condition, and/or texture condition inside    the enclosure. This is useful in graphical environments such as GUIs    and web pages/web browsers.

In another example of an enclosure, a circular or elliptical shape isdefined. For example, an elliptical enclosure can include only one wallthat is curved all the way around the ellipse. The inner and outer wallscan still be provided, as well as the rest of the other parameters. Thedimensions of the ellipse/circle can be provided as X and Y widths.Alternatively, an eccentricity parameter can define the eccentricity ofthe ellipse.

With enclosures, an issue often arises whereby the forces of theenclosure are desirable at one point in time but not at a differentpoint in time. For example, a button having an enclosure has anattractive force associated with the enclosure to assist the user inacquiring the button with the cursor. However, once the cursor ispositioned inside the enclosure, the user may want to be able to freelymove the cursor again without experiencing the enclosure wall forcessince the target button has already been acquired. Thus, in anembodiment, the forces associated with the enclosure are turned off oncethe cursor is moved inside the enclosure. In an embodiment, the forcemagnitudes can be weakened or adjusted to a lower level, e.g., ⅓ thenormal magnitude, to allow the cursor to easily exit the enclosure. Oncethe cursor has exited the enclosure, the forces are then turned back on.

In an embodiment, the forces associated with the enclosure can be turnedoff or weakened for a predetermined period of time to allow the user tomove the cursor out of the enclosure without hindrance, and then theforces can be automatically turned back on once the time has expired. Inan embodiment, the forces are turned off only when the cursor has notmoved for a predetermined period of time. This indicates that the userhas acquired the desired target. In an embodiment, the forces may beturned off when a command gesture, such as a button 15 activation, isdetected. These various embodiments can also be combined in otherembodiments as desired. The host or microprocessor can determine whenthe enclosure has been entered and exited by the graphical objectthrough the use of events, explained below.

Enclosures can also be defined to compensate for particularities of usercontrol of the user device. For example, up and down movement of a mousefor most users is much easier that left-to-right movement. An enclosurehaving equal strength forces on all walls will feel as if the left andright walls have stronger forces. Thus, the upper and lower walls can bedefined with stronger forces, e.g., two times the magnitude of the leftand right wall forces, to compensate for this effect. Such an enclosuretypically feels as if forces of all the walls are of equal strength.

FIG. 6 c illustrates another use of enclosures in accordance with anembodiment. Enclosures can be used with events to provide someenclosures with priority over other enclosures. The enclosure withpriority can change the characteristics of another enclosure, such asthe strength of forces of the walls of the other enclosure. For example,window 272 is associated with a force enclosure 274. The walls ofenclosure 274 surround the borders 275 of the window and have anassociated force to attract the graphical object to the border. Buttons277 are also included in window 272 which are near the border 275 andwhich may be selected by the user. Buttons 277 have their own enclosuresdefined in the interior of the buttons. The problem is that the forcesfrom the enclosure 274 and the forces from the button enclosuresconflict. A user may not be able to select the button easily due to thegraphical object being stuck between competing enclosure forces.

In an embodiment, an enclosure 276 can be defined to control the effectsof other enclosures. For example, the enclosure 276 occupies a regionaround buttons 277 and is not visible to the user. Enclosure 276 turnsoff the forces of enclosure 274 inside its borders and leaves the snapforces of the buttons 277 on. This allows the user to feel the snapforces of the buttons without conflict from the forces of the windowenclosure 274. Alternatively, the forces of enclosure 274 can be reducedinside enclosure 276 to an extent so that the user is not encumbered bycompeting forces. A similar type of enclosure can be used on sliders,where the moveable “thumb” of the slider can be provided with anenclosure similar to enclosure 276 surrounding it. The enclosuresurrounding the slider turns off or reduces forces from other enclosuresand allows the snap attractive force of the thumb to attract the userobject to the “thumb.” In addition, events may be used to determine whenthe graphical object enters the enclosure 276.

FIG. 7 illustrates a different effect useful in force feedbackapplications known as a texture in accordance with an embodiment. Atexture is a condition, similar to friction, which causes the userdevice or user's digit to feel as if it were traveling over a series ofbumps. FIG. 7 illustrates a distance vs. force graph in which a numberof bumps are defined with a coefficient C, a period T, and a deadbandDB. Bumps in the positive distance region are felt when the user deviceor digit is moved in a positive direction, and bumps in the negativedistance region are felt when the user device or digit is moved in thenegative direction. There is a separate height (coefficient) and periodof the bumps for the positive and negative directions along a particularaxis of the mouse 12 so that different textures can be felt depending onthe direction of the user device or digit. Directional textures can beused, for example, to achieve a velvet-like effect. The deadband valuecan apply for both directions. Textures are defined as a condition andcan include parameters such as:

-   -   Positive Coefficient, Negative Coefficient: Height (magnitude)        of bumps when travelling along the axis in the positive or        negative direction, respectively.    -   Positive Saturation, Negative Saturation: Period, in pixels, of        bumps (bump width plus deadband width) when travelling along the        axis in the positive or negative direction, respectively.    -   DeadBand: Percentage of period which is not occupied by a bump.

FIG. 8 illustrates a different force feedback effect useful in forcefeedback applications known as a “grid.” A grid is a condition thatcreates a 2-dimensional array of snap points 280, which are forcesattracting the user device to the point when the user device moves overthe point. Grids can be stand-alone conditions, but they can also beuseful when associated with enclosures. The geometrical attributes of agrid inside of an enclosure 282 are depicted in FIG. 8. Grids aredefined as a condition and can include parameters such as those below,also illustrated in FIG. 8.

Offset (O): If the grid is inside an enclosure, the Offset defines thehorizontal and vertical distance, in pixels, from the upper left cornerof the enclosure's deadband to the first snap point. If the grid isstand-alone, it defines the horizontal and vertical distance, in pixels,from the upper left corner of screen to the first snap point.

Positive Coefficient (C): Strength of snap points (jolts).PositiveSaturation (S): Saturation of snap points.DeadBand (DB): Vertical and horizontal spacing, in pixels, between snappoints.

Grids can also be implemented as one-dimensional forces, e.g., as tickmarks on a slider or any other graphical object when moved. The tickmarks would only be felt when the user device or user digit was movingin the defined dimension. For example, a slider can be defined with aone-dimensional grid so that, as the knob of the slider is moved, a snappoint is applied as the knob moves over each tick of the grid. The ticksthus have a spatial correlation with the distance that the knob hasmoved and inform the user of such.

Events

One structure used by the host computer in force feedback implementationis called an “event.” Events are notifications from the force feedbackdevice to application programs 202, 204, and 206 of one or more actionsor interactions that have occurred in the graphical environment whichhas been sensed by the force feedback user device. For example, an eventcan be any interaction of the graphical object in the graphicalenvironment with another graphical object. Thus, an event can be definedwhen the graphical object enters into an enclosure object (describedbelow) through a particular border. In an embodiment, an event can betriggered when the graphical object moves over a close box of a window,or the graphical object moves over a file icon and a button is pressed.In an embodiment, an event can also be defined when the graphical objectmoves within a specified range of a particular graphical object orregion. In an embodiment, an event may be defined with a temporalcomponent, e.g., an event is triggered when the graphical object remainsin a particular region for a specified period of time. Events aresimilar to button press triggers in that a condition is detected on theforce feedback device, independently of the host computer. However, thetrigger for the event is a graphical interaction rather than a physicalbutton press. In an embodiment, the trigger for the event may be agraphical interaction in addition to any user input into the userdevice.

An application program initially defines what the event is by creatingan event definition with the API 208 (FIG. 4). In an embodiment, theevent definition includes an event identifier, an application program(or window) identifier, information specifying actions that trigger anevent notification, and force effects, if any, that the event isassociated with. The application program, in an embodiment, assigns aseparate identification value to each event at the time of creating theevent, and keeps track of the created identification values in its ownlist. The API can enable, disable, or delete an event at the request ofan application program.

The translation layer receives and stores events when the applicationthat created the event is the active or background application program.Thus, events are included in the active context. The translation layer218 checks for events based on the position of the graphical object (oruser device) received from the force feedback device when the associatedapplication program is (or becomes) active. The translation layer 218then sends the event notification up to the context driver (or API),which sends an event notification to the application program, e.g.,sends a message to a window handler which the application programchecks. The event notification includes the event identifier so that theapplication program can check its own list and identify the event. Onceidentified, the application program initiates the appropriate responseas defined by the developer, e.g., running another application program,outputting a sound, displaying a new image, etc. Alternatively, theforce feedback device is loaded with the defined event similar to beingloaded with effects. The local microprocessor 130 can check for events(for active application programs) and send information back through thetranslation layer to the context driver to notify the applicationprogram.

For an application program to receive an event notification, theapplication's context must be active, i.e., the application must beactive or background. Thus, inactive applications will not receive eventnotifications. Some events can be defined to have priority over otherevents. For example, if a trigger is associated with an event in aforeground context as well as an event in a background context, theforeground context can be defined to have priority so that only theapplication program in the foreground receives an event notification. Inan embodiment, only the background application, or both the foregroundand background applications, can receive the event notification.

In an embodiment, both one-time event triggers and periodic eventtriggers are available. One time event triggers are triggers associatedwith discrete actions that have no significant duration, such asbreaking through an enclosure. Periodic event triggers are associatedwith conditions that have a finite duration, such as being inside thewall of an enclosure, or holding down a scroll button. During theduration of the condition which triggers the event (e.g., while thecursor is inside the enclosure), the API repeatedly sends eventnotifications to the application program according to a predeterminedperiodicity that can be set by the user or application program.

In an example embodiment, the application program creates an eventdefinition for an enclosure event. This is an event that occurs when auser moves the cursor into an enclosure. The event definition includes aunique event identifier, an identifier for the application program thatcreated it, actions that trigger the event notification (entering theenclosure), and any effects (such as the particular enclosure) which theevent is associated with.

An example of an event structure is provided below:

EVENT typedef struct { DWORD dwSize; HWND hWndEventHandler; WORD wRef;DWORD dwfTrigger; LPI_EFFECT pEffect; } EVENT;

-   dwSize: Size of the structure in bytes. This member must be    initialized before the structure is used.-   hWndEventHandler: Handle of window to which event notifications are    sent.-   wRef: 16-bit application-defined value to identify the event. This    value is defined by the application to identify events that it has    created. The application assigns a separate wRef value to each event    at the time of creation. Each subsequent event notification that the    application receives will be tagged with its associated wRef value.    When the application processes the window message that receives    event notifications, it will compare the wRef value to its own    internal list and take action accordingly.-   dwfTrigger: Flags specifying actions that trigger the event    notification.-   pEffect: Particular effect (e.g., enclosure), if any, in the GUI    that this event is associated with.

The following are examples of flags that can be included as triggers foran event.

ONENTERTOP, ONENTERLEFT, etc.: The mouse broke into the associatedenclosure by breaking through the top, bottom, left, or right wall, asspecified.ONEXITTOP, ONEXITLEFT, etc.: The mouse broke out of the associatedenclosure by breaking through the top, bottom, left, or right wall, asspecified.ONENTERANY, ONEXITANY: The mouse broke into or out of the associatedenclosure.INBOUNDTOPWALL, INBOUNDLEFTWALL, etc.: The mouse is in the top, bottom,left, or right wall, as specified, and attempting to break into theassociated enclosure.OUTBOUNDTOPWAL, OUTBOUNDLEFTWALL, etc.: The mouse is in the top, bottom,left, or right wall, as specified, and attempting to break out of theassociated enclosure.INBOUNDANYWALL, OUTBOUNDANYWALL: The mouse is in a wall and attemptingto break into or out of the associated enclosure.ANYWALL: The mouse is in a wall of the associated enclosure.ONSCROLL: The user is holding down the scroll button of the mouse.ONEFFECTCOMPLETION: The associated effect has completed.

The host computer or force feedback user device may also command theuser device to act as a different type of control device for someembodiments. For example, the force feedback user device may also beused in conjunction with game application programs, simulations, or thelike, in which a joystick or other type of controller is typically used.User device can be configured to work with such game applications in a“joystick mode” as if it were a standard force feedback joystick, or asif it were a non-force-feedback joystick. The user device may include aphysical switch or software switch provided in master application 206 orother program on the host computer to allow switching between thejoystick mode and any other mode capability of the user device.

The mouse user device 12 as shown in FIG. 1 b is not easily applicableto many applications intended for use with a joystick, since the mouseis a planar device having a free workspace while a joystick typicallyhas rotary degrees of freedom and springs to bias the joystick to acenter position. In an embodiment, when the mouse device 11 is used inthe joystick mode, spring forces are applied to bias the mouse 12 towardthe center of its workspace. These forces are applied by actuators 64and controlled by the local microprocessor 130 as background forces thatare always in effect, unless the device is switched out of joystickmode.

Force Feedback Device Implementations

In an embodiment, the force feedback user device includes a localmicroprocessor 130 that implements various processes to provide thefunctionality and features of the force feedback implementationdescribed herein. Various aspects of the force feedback deviceimplementation are described below.

Reference Frames

The force feedback user device as described herein needs the ability toknow the location of the graphical object on the display device. This isbecause the device must monitor the interaction of the graphical objectwith other objects in the graphical environment to determine whenassociated forces should be output by the user device. The forcefeedback device uses different reference frames, in an embodiment, todetermine where the graphical object is positioned on the display. Thereare three primary reference frames that are described herein: the device(or local) frame 30, the ballistic frame, and the screen (or host) frame28.

The device frame 30 is the frame of the physical workspace of the forcefeedback device. In an embodiment, the device frame 30 is defined withthe origin at the top left, the positive x direction to the right andthe positive y direction down. This frame 30 is used to describe theabsolute position of the end point of the mechanism in its workspace.The scale of “movement units” for the device frame 30 is arbitrary andfixed, although it should be a higher resolution than the resolution ofsensors 62 to guarantee no loss of sensor data. Furthermore, in anembodiment, the device position is scaled to reflect a 4:3 aspect ratio,which matches the typical computer monitor size, although other aspectratios are contemplated for different applications and display sizes.

In an embodiment, rate-control indexing is provided by the system. Thisallows the user to utilize the user device in reaching the limits of itsworkspace and still control the graphical object in the same direction.In brief, the central portion of the device workspace is designated asthe position control region, where a change in position of the userdevice in its workspace directly correlates to a change in position ofthe graphical object on the screen. The change in device position istranslated to a change in ballistic position based on the velocity ofthe device, according to a ballistics algorithm (described below).

An edge region of the device frame 30 is designated as the rate-controlregion, in which the mode of moving the graphical object changes fromposition control to rate control, where the graphical object continuesto move on the screen in the same direction while the user device is inthe rate control region. The rate control region boundary is accompaniedby a repelling force that pushes the user object towards the center ofthe device workspace. As the user pushes against this force, thegraphical object is moved in a speed proportional to the distance thedevice is displaced into the rate control region. Therefore, thegraphical object moves relative to device position in the rate controlregion, not relative to device motion. This mode may be thought of asmoving the position control central area of the device frame around thescreen using rate control. The size of the rate control region isspecified by a percentage. For example, a 400.times.300 device workspacehas a rate-control region of 10%. The graphical object position would bedetermined according to a position control paradigm when the user objectposition is read between the values of (x, y)=(20,15) and (380, 285) ofthe device frame, and according to a rate control paradigm if the userobject position is outside of that range.

The screen frame 28 is used to describe the position of the cursor (orother user-controlled graphical object) on the screen. It is definedwith the origin at the top left, the positive x direction to the rightand the positive y direction down. This frame is scaled corresponding tothe resolution of the display device 20 (or devices 20). For example, ifa display screen resolution is set to 1024.times.768 pixels, the xscreen position ranges from 0 to 1023 as the cursor moves to the rightand from 0 to 767 as the graphical object moves down. All coordinatecommunication between the force feedback device and the host computerare in terms of screen coordinates (normalized for device standards).

The ballistic frame (or “scaled frame”) is a frame created by the forcefeedback user device to determine the position of the graphical object.By applying a ballistics algorithm to the change in position of the userobject, the position of the graphical object is determined. Theballistic frame is defined as a multiple of the screen frame, i.e., theballistic frame has a higher resolution than the screen frame, but hasthe same aspect ratio as the screen frame. The ballistic frame iscreated by the force feedback device so that pixel changes smaller thanone screen pixel can be made to the graphical object position, which iscrucial when implementing the fast control loop rates demanded byforce-feedback. For example, if the control frequency is 1 kHz, and amonitor provides 1000 pixels across, a change of 1 pixel per controlcycle would cause the cursor to travel the entire screen width in onesecond. In rate control mode, such adjustments in the graphical objectposition may have to be made every control cycle, which is far too fastto allow control of the graphical object. Thus, a frame with higherresolution, the ballistic frame, needs to be defined to allow the forcefeedback device to change cursor position at its own control frequency,but which can be divided down to the screen frame size to slow down thegraphical object as it is moved across the screen. In addition, theaspect ratio of the ballistic frame is the same as that of the screenframe. This allows force effects on the device to spatially correlatewith graphical objects displayed on the display device. This isdescribed in greater detail below.

In an embodiment, the microprocessor makes transformations betweenframes to convert a device position to an eventual screen coordinate.When transforming from the device frame to the ballistic frame, themicroprocessor may use sensitivity. Sensitivity is the ratio of the sizeof the device frame to the size of the ballistic frame. A higher valueof sensitivity allows the cursor to traverse the screen with lessphysical motion of the user device, thus making it more difficult tocontrol the graphical object. A lower value of sensitivity allows moremotion on the device to cause less motion of the graphical object on thescreen, and is suitable for finer control. At low sensitivity values,the graphical object may only be able to reach the entire screen area byusing rate control mode.

In an embodiment, the graphical object can exactly move across theentire width of the screen as the user device or user's digit ismanipulated across the extent of the position control region of thedevice workspace, as long as the velocity remains in a middle range toprovide a one-to-two ballistic mapping (a one-to-one ballistic mappingis provided with velocities in the lower range, causing less movement;and a one-to-four ballistic mapping is provided with velocities in thehigh range, causing more cursor movement). As sensitivity is increased,the screen can be traversed in less than the entire position controlregion of the device workspace (assuming a middle range velocity). Atlower sensitivities, the entire screen cannot be traversed (assuming amiddle range velocity) without using the rate control mode.

Since the aspect ratios of the device frame and the ballistic frame maynot match, updating position changes in the ballistic frame may requirethat a change in device position (or a simulated change, when in ratecontrol mode) be added to the ballistic frame. If changes in mouseposition are always used, problems with different aspect ratios areeliminated. This is described in greater detail below.

Transforming from the ballistic frame to the screen frame is simpler.Once the ballistic position is known, the position of the graphicalobject in the screen frame can be found by dividing the ballisticposition by the known multiple. Thus, if the ballistic frame is definedto be 10 times the resolution of the screen frame, the ballisticposition is divided by 10 to obtain the screen position.

Position Reporting

Standard user device controllers, such as a computer mouse, areopen-loop devices which report position changes in its workspace to thehost computer and have no ability to know the current cursor position.The force feedback user device, in contrast, needs to know the locationof the graphical object on the display device of the host computer,since the user device must monitor the interaction of the graphicalobject with other objects in the graphical environment to determine whenassociated forces should be output.

There are two primary means for a force feedback user device to reportits position to the host and to know the screen position of thegraphical object. In an embodiment, the user device knows the locationof the graphical object by reporting (normalized) absolute coordinatesto the host computer to dictate the position of the graphical object.For example, the actual screen coordinates of the graphical object canbe determined by the user device and reported to the host, whereby thehost displays the graphical object at the reported coordinates. Thisallows the user device to know exactly where the graphical object is atall times, and to therefore have an accurate model of the screen for usewith outputting forces. However, any ballistic behavior of the graphicalobject must be calculated on the user device itself, creating morecalculation overhead for the local microprocessor 130. In addition, anabsolute reporting scheme is not compatible with traditional,non-force-feedback user devices, thus excluding a default mode in casethe force feedback functionality is not operational.

A second method provides a force feedback user device that reportsposition changes in its workspace and relies on the host computer tosend back the position information. This approach allows the hostsoftware to maintain the job of doing ballistics, but requires twice thecommunication as well as a need for the mouse to predict intermediatevalues of position while waiting for the next host position update,since the reporting occurs at a slower rate (e.g., about 50-60 Hz) thanthe haptic control rate (e.g., about 1000 Hz). This embodiment isdescribed in greater detail with respect to FIG. 13.

In an embodiment, the processes are implemented as firmware inintegrated circuits and memory provided on the force feedback device.Alternatively, the processes can be implemented as hardware or software,or a mixture of these. The processes themselves can be implemented usingprogram instructions or code stored on or transferred through a computerreadable medium, such as a memory chip, circuit or other device;magnetic media such as hard disk, floppy disk, or tape; or other mediasuch as CD-ROM, DVD, PCMCIA cards, etc. The program instructions mayalso be transmitted through a channel to device 11 from a differentsource.

FIG. 9 is a block diagram illustrating an embodiment 290. In thisembodiment, both relative and absolute position reporting modes areused. As referred to herein, relative position reporting means that theforce feedback device reports only changes in position of the userobject 12 to the host computer, i.e., a difference between a newposition and the last reported position, while absolute positionreporting means that absolute coordinates are reported to the host fromwhich the host can directly display the graphical object. Most forcefeedback controllers use absolute position reporting since the positionof the graphical object on the screen must be known to implement forces.

With absolute position reporting, the host computer cannot detect theforce feedback device as a standard input controller, such as a mouse,touch screen, or other user device. If relative position reporting wereprovided, the host computer can detect the force feedback user device asa traditional non-force-feedback user device. This is desirable whenforce feedback functionality is not active to allow the user to selectoptions within the operating system, such as during startup of thedevice before force feedback drivers have been loaded or duringemergencies when force feedback is not enabled.

In an embodiment, the system includes a relative positioning mode whenthe force feedback user device is powered up and/or when the hostcomputer 18 first detects the force feedback device. In this mode, theuser device provides a device delta position to the host as shown inFIG. 9. The host expects to receive a delta value upon startup, and thussimply processes the delta position normally. This allows the user touse the device for normal positioning of the graphical object in agraphical environment before force functionality has been initialized.The microprocessor does not need to model the graphical object positionin this stage of the process because no force feedback functionality hasyet been enabled on the user device. However, once the force feedbackfunctionality is enabled, e.g., the force feedback driver software hasbeen enabled on the host computer, then an absolute position is reportedto the host in place of the delta position relative position reporting.In addition, ballistic parameters and the host screen size are sent tothe user device to allow the user device to determine the absoluteposition of the graphical object. Force feedback commands are also sentto the device in this stage when appropriate.

FIG. 10 illustrates a process 300 for providing force feedback inaccordance with an embodiment. In an embodiment in which the user deviceincludes a mechanical portion as described above, current joint angles302 are measured by sensors of the force feedback device. For example,in the embodiment of FIGS. 1 and 2, the joint angles between members 44,46, 48, and 50 are determined based on the current position of theencoder arc as sensed by the emitter and detector of sensor 62. Thejoint angles can be determined since the precise measurements of themembers of linkage 40 are known and are in a known position relative toeach other based on the sensor 62 reading. In a process box 304, thecurrent joint angles are normalized based on the sensed range of themouse 12 in its degrees of freedom. Since the exact range of the mouse12 (or other user object) may not be precisely known due to variationsin the device or other irregularities, the normalization process uses asensed range of motion to determine and assign physical angles betweenthe members of the linkage 40.

For example, the minimum sensor value that has been detected so far isconsidered to be one limit of the motion of the mouse, and the maximumsensor value detected so far is considered to be the other limit to themouse range. In successive iterations of the process 300, the mouseeventually will be moved to the physical limits of the device and theangle values will be updated accordingly. This tends to produce moreaccurate results than an assumed range of motion, since manufacturingtolerances of the components of device 11 provide slightly varyingdimensions, link lengths, and angle ranges that cause discrepancies fromdevice to device. In an embodiment, the force feedback device(microprocessor 130) can control the actuators of the device to move themouse 12 to all the limits of the mouse 12 in a short start-up procedureto determine the actual angle ranges. It should be noted that although amouse is referred to in relation to the above description, any otherdevice can be utilized instead of computer mouse without departing fromthe inventive embodiments discussed herein.

In the embodiment described in FIGS. 1 and 2, the normalized jointangles 306 resulting from box 304 are then processed by applyingkinematics in box 308. Kinematic equations, as is well-known to thoseskilled in the art, are used to derive a position of a desired point ona mechanical linkage or other apparatus from known lengths of links inthe linkage and the current angles between those links. For example, aCartesian position of a point on the linkage in a Cartesian (or othertype) coordinate system can be determined using geometrical equations. Acurrent device position 310 is determined from the kinematics of box308, which is the position of the mouse 12 in its workspace.

The current device position 310 is used, with a previous device position312, to determine a device delta position 314. The previous deviceposition 312 was the current device position 310 in a previous iterationof process 300; this is indicated by the dashed line 311 between currentand previous device positions. The device delta position 314 is simplythe difference between the current device position 310 and the previousdevice position 312. In box 316, the microprocessor 130 sends thedetermined delta position 314 to the host computer 18 over bus 120. Thisis a relative position reporting step and is only performed if thesystem 10 does not yet have force feedback functionality, e.g., atstartup when the force feedback drivers are being loaded. The hostcomputer uses the delta position to position the graphical object on thescreen of display device 20. For example, the host computer adjusts theposition of a graphical object by the delta amount to achieve a newposition of the graphical object in the screen frame 28. During relativeposition reporting mode, the delta position 314 is sent to the host at aslower rate than the rate of process 300; therefore, in some iterationsof process 300, the delta need not be reported to the host. This isbecause the host only needs to receive a cursor delta position at therate of displaying the cursor on the screen, which typically is on theorder of 60 Hz or every 20 ms. The microprocessor 130, on the otherhand, must control forces at a much higher rate (1000 Hz or above) andthus needs to execute process 300 much faster.

In the embodiment in which the device is in relative position reportingmode, the process iteration is complete after the delta position 314 isreported to the host. The process then returns to measure joint angles302 and start the next iteration when triggered to do so. Themicroprocessor does not need to model the cursor position in this modebecause no force feedback functionality has yet been enabled on thedevice 11.

Relative position reporting mode is then exited and absolute positionreporting mode is entered when the appropriate software drivers or otherprograms are loaded into the memory of host computer 18 to allow thehost to recognize, interface with, and command the force feedbackinformation to the user device 11. In absolute position reporting mode,the delta position 314 continues to be determined as described above.However, the delta position is no longer reported directly to the hostcomputer, but is instead used in a ballistics process to determine aballistic delta, described below.

In absolute position reporting mode, the force feedback device 11 alsodetermines a velocity of the user object for use in the present processin addition to or alternative to the joint angle processing describedabove. A current joint velocity 320 is first measured. In the preferredembodiment, this is accomplished by a dedicated circuit or device thatdetermines a velocity from the sensed positions reported by sensors 62,e.g., the sensed positions of members 44 and 48. For example, a “hapticaccelerator” device can include a circuit dedicated to calculatingvelocity from multiple sensed positions and time periods between thesensed positions. Alternatively, velocity sensors can be used todirectly measure velocity of joints or members of the device.

The current joint velocity 320 is used in a process box 322 which useskinematics in determining device velocity. As described above,kinematics are well-known equations for determining position, and alsoare used to determine a velocity of a desired point of a linkage. Thus,the kinematics can be used to determine the velocity of the mouse 12,which is referred to as the “device velocity” herein. The current devicevelocity 324 resulting from box 322 indicates the current velocity ofmouse 12.

In an embodiment, the ballistics box 328 uses a ballistic algorithm orsimilar method to model or predict a position of the graphical object inthe screen frame 28 based on the position and velocity of the userdevice. This modeling is accomplished to allow the local microprocessor130 to determine where the graphical object is positioned in theballistic frame. Using the spatial layout of graphical objects that isstored locally in memory 134 and which is continually updated by thehost, the local microprocessor determines when the graphical object isinteracting with a graphical object and when to output forces associatedwith the interaction. The local determination of the graphical objectposition allows the microprocessor to know the position of the graphicalobject for the determination of forces.

In the ballistics box 328, the local microprocessor uses a ballisticsalgorithm to determine a “ballistic delta” for the graphical object,(i.e., a change in position of the graphical object measured in theballistic frame), and an embodiment. A ballistics algorithm is a methodof changing the position of the graphical object so as to break thedirect mapping between the user device's position and the graphicalobject position to allow the user greater control over the position ofthe graphical object. For example, most ballistics algorithms vary theposition of the graphical object based on the velocity of the userdevice in its workspace. The faster the velocity of the user device, thegreater the distance that the graphical object is moved on the screenfor a given distance the user device is moved. This allows slow motionsof the user device to finely control the graphical object, while fastmotions coarsely control the graphical object and allow the graphicalobject to be moved quickly across the screen. Other algorithms can alsobe used which are similar to ballistics in that graphical objectposition is altered to allow fine graphical object control in situationswhen needed and to allow coarse graphical object control when needed.

In an embodiment, the microprocessor 130 uses a ballistic algorithm thatprovides two velocity thresholds to determine how far the graphicalobject is to be moved or manipulated. This is described in greaterdetail below in the method of FIG. 11. Other ballistics algorithms maybe used in other embodiments; e.g., a continuous algorithm that adjustsgraphical object position continuously along a range of mouse velocity.

In an embodiment, the microprocessor uses several types of data todetermine the ballistic delta of the graphical object, including thedevice delta position 314, current device velocity 324, and hostballistic parameters 330. The host ballistic parameters 330 are receivedby the microprocessor 130 from the host at some point before the process300 starts, or when the ballistic model changes, and are stored in localmemory 134 to be accessed when required. Such parameters can include thedistance to move the graphical object for each threshold range of devicevelocity, the velocity values of the two thresholds, and any conditionsconcerning when to apply the ballistics algorithm. If the host changesballistics algorithms, notification can be sent to the microprocessor130 and new parameters stored in local memory 134. The microprocessor130 examines the current device velocity 324 and the parameters 330 todetermine in which threshold range the mouse velocity currently isplaced, and then multiplies the device delta position by the appropriateamount (e.g., 1, 2, or 4, depending on the velocity). This results inthe ballistic delta 332, which is provided in terms of the ballisticframe.

Changes in position of the user device (i.e., delta positions 314) areused in the absolute position reporting mode, because this allowsproblems with changes to screen aspect ratios and resolutions to bealleviated. In an embodiment, an indexing function of the device isimplemented which may cause the ballistic delta 332 to be calculateddifferently in a rate control mode.

The microprocessor then proceeds to a process box 334, in which acurrent ballistic location (or “scaled position”) 336 of the graphicalobject is determined in the ballistic frame. To accomplish this, themicroprocessor uses the ballistic delta 332, and the previous ballisticlocation 338. The previous ballistic location 338 was the currentballistic location 336 in an earlier iteration of process 300, asindicated by dashed line 337. Previous location 338 can be stored inlocal memory 134 and then discarded after being used to determine thecurrent ballistic location. At its simplest level, the current ballisticlocation 336 is determined in box 334 by taking the previous ballisticlocation 338 and adjusting that location by the ballistic delta 332.

The current ballistic location 336, however, cannot be directlytransformed to a screen pixel position in the screen frame. Box 342 usesthe current ballistic location 336 and the host screen size 340 todetermine the screen pixel location. In an embodiment, the host screensize 340 is the current pixel resolution and/or aspect ratio beingdisplayed on the display device 20 of the host computer 18. For example,the host may be displaying a screen resolution of 800.times.600 pixels,1024.times.768 pixels, or 1280.times.960 pixels. Also, in someembodiments, the host may be displaying a different aspect ratio. Forexample, a typical computer monitor aspect ratio is 4:3; however, if twomonitors are connected to provide a continuous screen space, the aspectratio becomes 8:3, i.e., double the size of a single screen in onedimension. The host screen size 340 can be received by the device 11when the ballistic parameters 330 are received, or at any other time(and/or when sensitivity information is received, if applicable).Different resolutions or aspect ratios have different numbers of pixelsin respective dimensions of the screen and thus cause displayed objectsto appear in different areas of the screen. For example, if a graphicalobject is centered in the screen at 640.times.480 resolution, the objectwill appear in the upper left quadrant of the screen when the resolutionchanges to 1280.times.960.

The change in resolution or aspect ratio can create a problem with forcesensations created by the force feedback device. The device has beensent a spatial layout of objects from the host that indicates thecoordinates of an object on the screen. When the screen size is changed,and the device is not informed, the device will output forcescorresponding to the object as it appeared in the old screen size. Thiscan cause large discrepancies between where an object is displayed andwhere the forces of an object are encountered in the device workspace.For example, if an icon is at coordinates (64,48) on a 640.times.480resolution screen, the device knows that the icon is positioned 10% ofthe entire screen dimension away from the top and left borders of thescreen workspace. The device accordingly outputs associated forces whenthe user device is moved into the region of the device workspacecorresponding to the icon, i.e., 10% of the workspace distance from thetop and left borders of the workspace. However, after a resolutionchange to 1280.times.960, the icon is displayed 5% of the screendimension away from the top and left borders. The device, however, wouldoutput forces based on the old position of the icon, which no longercorresponds with the position of the icon. Thus, the device needs to betold the new screen resolution so it may provide forces at theappropriate region of the device workspace.

Similarly, a change in aspect ratio can cause distortion in the forcefeedback sensations output by the user device. For example, a circleobject is displayed on a 4:3 aspect ratio screen and has a force wallsurrounding its shape. The circle will feel like a circle on the deviceif the ballistic frame also has an aspect ratio of 4:3. However, ifanother monitor is activated so that the screen has an 8:3 aspect ratio,the circle object will feel like an oval if the screen is directlymapped to the device frame, i.e., if the device frame is “stretched” tofit the screen frame. The use of delta positions 314 prevents thisdistortion. Since only changes in the mouse position are used todetermine graphical object position, and not an absolute mouse positionin the device workspace, the graphical object is positioned withoutdistortion. However, the new aspect ratio still needs to be communicatedto the force feedback device since there is a larger area in which thegraphical object may be positioned. For example, in a 4:3 aspect ratio,the microprocessor might clip a pixel location to the edge of the screenif that location were beyond the edge of the displayed range. However,on an 8:3 aspect ratio, the location should not be clipped since thedisplayed range is larger.

To prevent distortion caused by a change in screen size, the ballisticlocation 336 of the graphical object is divided by a divisor that takesthe host screen size into account, in an embodiment. For example, if theballistic frame is 10 times the resolution of the screen frame, then thedivisor is 10. Therefore, the ballistic location 336 would be divided by10 to achieve a screen pixel position of the graphical object if thehost screen size had not changed. If the host screen size did change,then the ballistic frame is resized to take the new screen size intoaccount. In an embodiment, the screen pixel position is equal to thescreen size times the ballistic location divided by the ballistic framesize (i.e., the divisor). Thus, if screen resolution on the host hasdoubled, the divisor would halve, and the screen pixel location woulddouble. The host can also send sensitivity information to the device. Ifa new sensitivity value is received, the ballistic frame is resizedaccordingly; as sensitivity is increased, the ballistic frame size isdecreased.

Once the screen pixel location is translated from the ballistic position336, the screen pixel location is then typically normalized to astandardized range of numbers since communication software and deviceshave standard values to receive. The normalized screen pixel position346 is then sent to the host computer as an absolute position in box350, e.g., as coordinates for display on the screen. The host receivesthe screen pixel location 346 and displays the graphical objectappropriately.

In box 344, the local processor 130 uses the (preferably non-normalized)screen pixel location (and other data) to determine whether any forcesshould be output, and outputs those forces. For example, the localmicroprocessor checks the spatial layout of graphical objects which isstored in local memory 134 and determines whether the screen pixellocation intersects or contacts any of those objects. If so, there maybe a force associated with the interaction, depending on the graphicalobject intersected. Such a force would be calculated based on a varietyof data. Such data may include the device velocity 324, the deviceposition 310, the elapsed time from an event, the distance from adifferent graphical object in the graphical environment, a history ofpast mouse motion, etc. Once box 346 is complete (and any other desiredboxes in the provision of forces, which is not detailed herein), theprocess when needed returns to the measuring of joint angles 302 andjoint velocity 320 to begin another iteration of the absolute positionreporting process.

In an embodiment, no error correction is required, because the forcefeedback device 11 is operating in absolute mode when the pixel positionis determined. The host thus no longer receives a change in position(delta) of the user object, but receives an absolute coordinate in acoordinate system. The host thus does not need to perform any othersignificant processing (such as ballistics) to display the graphicalobject, i.e., the local microprocessor solely performs the necessaryadjustments, such as ballistics, and reports the adjusted position tothe host computer. Thus, the host and local microprocessor have the sameposition value for the graphical object, and no opportunity exists forthe two positions to diverge.

FIG. 11 illustrates a flow diagram of a method in accordance with anembodiment. In particular, the method is for implementing enhancedgraphical object control and indexing in the absolute position reportingmode. This method provides ballistics for graphical object positioningas well as indexing functionality. In an embodiment, the user deviceworkspace includes a central position control region in which the userdevice controls the graphical object according to a position controlscheme. An isometric edge region is provided around the position controlregion of the user device workspace. When the user device is moved intothe isometric region, a resistive force is applied to the user deviceand the graphical object is moved according to a rate control method,where the speed of the graphical object is proportional to the amount ofpenetration into the isometric region. This allows a user to move thegraphical object throughout the screen frame without causing the user toexperience disconcerting interruptions due to the mouse colliding withphysical limits to the mouse workspace.

The method 360 is similar to the indexing methods described above inthat a position control region and isometric edge region are provided inthe user device workspace. Method 360 begins at 362, in which raw sensordata is read using the sensors 62. In box 364, the joint positions oflinkage 40 are calculated from the raw sensor data in an embodimentwhich utilizes a mechanical portion as shown in FIGS. 1 and 2. In box366, position kinematics are used to determine the device position(position of the user object) in the device frame, as explained abovewith reference to FIG. 10. The last device position is stored in localmemory 134 in box 368, and a delta device position is determined in box370 using the current device position resulting from box 366 and thelast device position stored in box 368. Thereafter, the system andmethod checks whether the device position is in the position controlregion of the device frame (372). If not, the user device is in theisometric region and is in rate control mode, where a resistive force ispreferably output on the user device opposing movement into the region.In box 374, the rate control delta is calculated to be the differencebetween the device position and the rate control edge; this provides thedistance of penetration into the isometric region. In box 376, the rateof graphical object movement is equal to the maximum rate times the ratecontrol delta divided by the rate control edge width (i.e., the width ofthe isometric region). The rate control delta divided by the ratecontrol edge width provides the percentage of penetration of the userdevice into the isometric region, which is designated to be the samepercentage of the maximum rate of the user device. The maximum rate canbe set by the developer in an embodiment, such as 1000 ballistic pixelsper second or 8000 ballistic pixels per second. In box 378, theballistic position (i.e., the position of the graphical object in theballistic frame) is set equal to the old ballistic position plus therate determined in box 376. The process then continues to box 394,detailed below.

If the device position is in the position control region in box 372,then a position control method is followed. Boxes 384-392 implement aballistics algorithm to provide enhanced graphical object control to theuser of device 11, in an embodiment. This algorithm is generally asfollows. If the current device velocity is below the first threshold,the graphical object is moved by the smallest amount, e.g., a one-to-onemapping between device movement units and ballistic pixels. If thecurrent device velocity is between the first and second thresholds, thegraphical object is moved double the amount of pixels as it is movedwhen user device velocity is under the first threshold, e.g., aone-to-two mapping between device and ballistic frames. If the userobject velocity is above the second threshold, the graphical object ismoved double the amount of pixels as between the first and secondthresholds (a one-to-four mapping between device and ballistic frames).In other embodiments, more or less thresholds, or a continuous function,can be used. Also, the host may change the ballistics algorithm in avariety of ways and indicate the changes to the device 11 usingballistic parameters 330 as described for FIG. 10.

Thus, in an embodiment, box 384 checks the device velocity so that aballistic position may be determined. The device velocity is determinedin boxes 380 and 382, where box 380 determines joint velocities from thesensor data read in box 362 and device velocity is determined from thejoint velocities and velocity kinematics in box 382, as described abovein FIG. 10. In box 384, this device velocity is compared to theballistic thresholds used in the ballistics algorithm. If the devicevelocity is within the first ballistic threshold in box 384, then box386 determines the new ballistic position to be the old ballisticposition plus the delta device position (from box 370). The process thencontinues to box 394, detailed below. If the device velocity is greaterthan the first ballistic threshold, then in box 388 the device velocityis compared to the second ballistic threshold. If the device velocity isbetween the first and second thresholds, then in box 390 the newballistic position is set equal to the old ballistic position plus twicethe delta device position. This causes the graphical object to move at afaster rate, based on the velocity of the user device. The process thencontinues to box 394, detailed below. If the device velocity is greaterthan the second ballistic threshold in box 388, then in box 392 the newballistic position is set equal to the old ballistic position plus 4times the delta device position. The process then continues to box 394.

In box 394, the screen position of the graphical object is set equal tothe ballistic position as determined in one of boxes 378, 386, 390, and392 divided by a divisor. The divisor is known from the resolution ofthe ballistic frame and the resolution of the screen frame, as follows.In box 396, the screen resolution (or size) is obtained from the host.In box 398, the divisor is set equal to the ballistic range (orresolution) divided by the screen resolution as detailed above.

After the screen position is determined in box 394, it is sent to thehost computer as shown in box 399 and used in the display of thegraphical object in the screen frame. The device 11 also determines andoutputs forces, as detailed in FIG. 10. It should be noted that theforces on user device may be determined or calculated differently inrate control (isometric) mode than in position control mode.

FIG. 12 is a block diagram of a method for reporting positions to a hostcomputer from a force feedback device in accordance with an embodiment.Embodiment 400 provides relative position reporting from the forcefeedback device 11 to the host computer 18. This is advantageous in thatthe force feedback device is detected by the host as a standardnon-force-feedback, where the host computer can use standard drivers andother software. This allows the user to control GUI functions even whenforce feedback is not currently active, such as during startup of thedevice or during failures of the force feedback functionality. Inaddition, a relative position reporting process is more appropriate thanan absolute position reporting process for such interface devices astrackballs, in which an unrestricted workspace and range of motion isprovided for the user object 12. For example, the spherical ball in atrackball controller, dexterous glove, or wireless game controller canbe moved in any direction for any number of revolutions and is not assuitable for use with an absolute coordinate system as a device having aworkspace with known limits.

Embodiment 400 provides a device delta position to the host computerfrom which the host computer determines a graphical object screen pixelposition. The host computer sends graphical object position updates tothe force feedback device to provide the graphical object position whichthe device needs to determine when forces are output and to determinethe force characteristics. However, since the host graphical objectposition updates occur at a much slower rate, the device must model thegraphical object position on its own to be able to output forces betweenhost graphical object updates. Thus, the host sends ballistic parametersand screen size information so the device can model the graphical objectposition. When the device receives an update, it corrects for any errorbetween its modeled position and the actual graphical object position,as detailed below. The host also sends force feedback commands to thedevice to indicate which forces are to be output.

FIG. 13 illustrates a process 410 for implementing the embodiment 400shown in FIG. 12. A difference between method 300 and method 410 of thefirst embodiment is that the host independently determines a screengraphical object position in parallel with the microprocessor modelingthe graphical object screen position. Since the host controls the actualposition of the graphical object, its screen position is considered the“correct” or actual graphical object position, while the device modeledvalue is only approximate. In an embodiment, the host continually sendsthe actual screen position to the device so that errors in the modeledscreen position can be corrected and so the forces can be correctlycorrelated with the actual graphical object position.

In an embodiment in which the user device utilizes a mechanical portion,the current joint angles 302 are measured and normalized in box 304, andthe normalized joint angles 306 are processed with kinematics in box 308to provide a current device position 310. The current device position310 and previous device position 312 are used to determine a deltaposition 314, which is reported to the host computer 18 in box 316 overbus 120. Unlike the process 300, the delta position 314 is alwaysreported to the host computer, i.e., there is no absolute position toreport at any stage of the process. The host computer uses the deltaposition to position the graphical object on the display device 20. Forexample, the host computer adjusts the position of the graphical objectusing the delta amount and its own ballistics algorithm to achieve a newposition of the graphical object in the screen frame 28. The deltaposition 314 is sent to the host at a slower rate than the rate ofprocess 410. Therefore, in some iterations of process 410, the deltaneed not be reported to the host, because the host only needs to receivea graphical object delta position at the rate of displaying thegraphical object on the screen, which typically is on the order of 60 Hzor every 20 ms. The microprocessor 130, on the other hand, computesforces at a much higher rate (1000 Hz or above) and executes process 300much faster.

The current joint velocity 320 also is read in process 410 (both thejoint angles 302 and the joint velocity 320 are always measured in thecurrent process). Kinematics are applied in box 322 to obtain thecurrent device velocity 324, and ballistics are applied in box 328 usingthe device velocity 324, ballistic parameters 330, and delta position314 to obtain a “modeled” pixel delta 332. The pixel delta 332 is amodeled or predicted value in this embodiment, because the host hasdetermined the actual screen position using its own ballistics algorithmand using the delta position 314 provided by the device. Themicroprocessor 130 preferably uses in box 328 the same ballisticsalgorithm used by the host computer in its determination of graphicalobject position.

In box 412, a current pixel location is modeled using the modeled pixeldelta 332, a previous pixel location 416, and the host screen size 340,resulting in a current pixel location 414. Box 412 is similar to box 334of process 300 in that pixel delta 332 and the previous pixel location416 are used substantially similarly. The host screen size informationcan be used to determine whether the determined pixel location should beclipped or not. For example, if the aspect ratio has changed from 4:3 to8:3, double the amount of area is provided. If the pixel location is onethat would normally extend beyond the edge of a 4:3 screen area, itwould be clipped to the border of the 4:3 screen, but may not be clippedif it is still within the 8:3 area. In one embodiment, if the change inscreen size is to a different resolution, the current pixel location canalso be adjusted depending on the change in resolution. For example, ifscreen resolution has been adjusted from 640.times.480 to1280.times.960, then the pixel delta can be doubled before it is addedto the previous pixel location.

Box 412 is different from box 334 in FIG. 10 in that a current pixelerror 418 is included in the determination of the current pixel location414. The pixel error 414 is the error between the graphical objectposition modeled by the local microprocessor and the actual graphicalobject position as determined and displayed by the host computer.Although the microprocessor is typically applying the same ballisticsalgorithm and screen size adjustments to the graphical object pixellocation as the host computer, errors can be introduced to cause the twographical object location calculations to diverge. For example, in someembodiments, the host computer can “clip” the received delta position314, i.e., ignore the delta position so that the graphical object staysat a constant position. Such a technique is used, for example, toprovide realistic obstruction force sensations when forces are limitedin magnitude. Additionally, error can be caused by an applicationprogram on the host computer moving the graphical object based on theprogram's own criteria and completely independently of the forcefeedback device. Further, error can be caused by the host computerswitching to a different ballistics algorithm, where a delay existsbetween the host using the new ballistic algorithm and the forcefeedback device receiving the parameters for the new algorithm.

The current pixel error 414 is determined by examining the current pixellocation 414 at the time of reporting the delta position 314 and acurrent screen position of the graphical object that has been receivedfrom the host computer. The host computer periodically reports a currentscreen position 420 of the graphical object to the microprocessor 130,where the screen position is based on the delta position 314 reported tothe host in box 316. The reporting of the screen position by the host istriggered by receiving the delta position 314 from the device, and themicroprocessor recalculates pixel error when it receives the host screenposition 420.

In an embodiment, the microprocessor compares the received screenposition 420 with the modeled current pixel location 414 at the time ofreport 316 (it should be noted that the current pixel location 414 mayhave a previously-determined error correction value included in itsdetermination, as explained below). If the two locations are the same,no error exists between the two graphical object positions, and themicroprocessor uses the current pixel location 414 as the location ofthe graphical object in its force calculations and force outputs. If thetwo locations are not the same, error exists between the two graphicalobject positions and the current pixel location 414 must therefore becorrected. An error correction value is determined as the differencebetween the current pixel location 414 and the host screen position 420.This value, when added to (or subtracted from) the pixel location 414,will cause the two graphical object locations to be the same.Alternatively, if the microprocessor knows the error, then the currentpixel location can be adjusted to provide no error.

In an embodiment, the screen position 414 from the host is reported tothe microprocessor 130 at a slower rate than the rate of process 410,since the force feedback process must be much faster than the displayingprocesses of the host computer. There may therefore be severaliterations of process 410 before the screen position 420 is reported bythe host to the microprocessor 130. Therefore, once the error correctionvalue is determined after one report of screen position 420, that samevalue is continually used in iterations providing the determination ofcurrent pixel location 414 until the next screen position 420 isreceived from the host.

In an embodiment, it is desirable to “smooth out” the correction of theerror in pixel location 414 by only incrementally changing the pixellocation 414 in each iteration of process 410. For example, if thecurrent pixel location 414 were adjusted with the entire errorcorrection value at one time, this might cause a large change in thepixel location 414. If forces output on the user object are based on thecurrent pixel location 414 of the graphical object, then the user maynotice the correction of the error by feeling a small jump ordiscontinuity in the forces applied to mouse 12 after the error iscorrected. This is because the forces were determined based on a pixellocation having a large error at one point in time, and then the forcesare based on the corrected pixel location at the next point in time.Thus, the difference in pixel locations in each iteration is preferablysmoothed by only partially correcting the current pixel location 414.With such smoothing, the user does not feel as large a discontinuity inforces when the error in pixel position 414 is corrected. The errorcorrection rate is preferably based on the desired system modeling. Theerror can be corrected at a rate so that there is no error between thepixel location 414 and the host screen position 420 by the time a newhost screen position 420 is received (which may provide a new error, ofcourse). Or, the error can be corrected at a rate that spans multiplereceived host screen positions 420. Typically, the correction of errorsis desired to be as slow as possible to maintain high fidelity of forcesto the user, yet the error must be corrected fast enough so as not tocreate a discontinuity between what the host is displaying on thedisplay device and the forces the user feels on the force feedbackdevice.

FIG. 14 is a diagrammatic illustration of an example of the pixellocation error correction process explained above. The line 426represents a time scale, and each vertical mark represents anotheriteration of process 410. At the first iteration, the pixel location 414is determined to be 192 (assuming a single dimension of user objectmotion) and the error 418 between the pixel location and host screenposition is 0. At the next iteration, the pixel location is 200, and thedelta position of 4 is reported to the host as in box 316. Over the nextiterations, the pixel location is determined to be 200, 201, and 202. Atthe sixth iteration, the pixel location is determined to be 203; inaddition, the host reports a host screen position of 201. Since thisscreen position was determined based on the delta position 314 reportedto the host in the second iteration, the error is measured between thepixel location determined at the second iteration (200) and the hostscreen position received at the sixth iteration (201). The error valueof 1 is then included in the current pixel location, so that the pixellocation of 203 is corrected to be 204 based on the error value. Thisexample demonstrates that the pixel location 384 determined by themicroprocessor 130, when error exists, is corrected a few iterationsbehind the actual screen position determined by the host. Themicroprocessor 130 locally stores the pixel location 414 that wasdetermined in the iteration when the delta position 314 was reported tothe host so that the error correction value can be accuratelydetermined. It should be noted that, in the example of FIG. 14, theerror value determined at the sixth iteration continues to be addedindefinitely. Thus, if further errors are indicated at later hostreports, the new error value must be added to the old error value.

FIG. 15 is a diagrammatic illustration of the error correction valueincreasing over time. Initially the error correction value is 0 and then1, as shown in FIG. 15. Over the next host report of the screenposition, the current error is 2 pixels, which is added to the previouserror correction value of 1 so that the total error correction value is3. Likewise, in the next host report, the current error is 3 pixels, sothat the total error correction value is 6. In one embodiment, if a verylarge error accumulates, the microprocessor can make the error 0 in oneiteration regardless of the effect on the user.

FIG. 15 also demonstrates the smoothing or incremental correction of themodeled pixel location in accordance with an embodiment. In theembodiment shown in FIG. 15, an error of 2 pixels exists at theiteration 428. The value of 2 is not added to the previous error valueof 1 immediately, since that may cause a force discontinuity to theuser. Instead the 2 pixels are added gradually to the error correctionvalue over the next few iterations. For example, the actual value (AV)added to the pixel location 414 is 0 for the first two iterations afteriteration 428, then AV=2 for three more iterations, until the full valueof 2 is added in following iterations.

Referring back to the embodiment in FIG. 13, once the current pixellocation 414 is determined, the local processor 130 uses that locationin box 424 to determine whether any forces should be output and, if so,outputs those forces. It should be noted that most forces are determinedpreferably based on the velocity of the user object in its workspace,not the velocity of the graphical object in the graphical environment.Thus, these forces will always feel the same to the user, regardless ofany error between the modeled pixel location 414 and the host screenposition 420, since they are independent of screen velocity. However,some forces are dependent on interactions in the graphical environment,and will be affected by a graphical object position error. Likewise, theability to initiate or discontinue forces based on interactions in thegraphical environment is also affected by graphical object positionerror. Once box 424 is complete (and any other desired boxes in theprovision of forces, which is not detailed herein), the process returnsto the measuring of joint angles 302 and joint velocity 320 to beginanother iteration of the process, and an embodiment.

Force Feedback Features as Implemented in One or More Embodiments

In the above described embodiments, the host computer performs suchtasks as the implementation and display of the graphical environment andgraphical object, the implementation of multi-tasking applicationprograms, the sending of high-level force feedback commands to the forcefeedback device, and the reception of position (or other) data from theforce feedback device indicating the user's desired interaction with thecomputer-implemented environment. The force feedback device, in anembodiment, performs such tasks as reading the position of the userobject, receiving high level force feedback commands from the hostcomputer, determining whether a force should be output based onconditions sent from the host computer, and implementing and outputtingthe required forces when necessary. However, it should be noted thatsome tasks may be implemented in either or both of the host computer andthe force feedback user device.

A force feedback feature that can be implemented in either or both ofthe host and the force feedback device is “clipping.” Clipping is theselective omission of reported position data when displaying thegraphical object or other user-controlled graphical object to cause asensory effect on the user in certain circumstances. Such circumstancesinclude those where displaying the graphical object in correlation withthe precise user object position would not provide a desired effect. Themost common case for clipping is when the graphical object encounters asurface or object in the graphical environment which is desired for theuser to experience as a hard obstruction or wall. The wall is desired toprohibit further movement into the surface or object; however, due tolimitations in hardware, a strong enough force cannot be output on theuser object to actually prevent motion into the wall.

Clipping can be easily performed by the force feedback device to easecomputational burden on the host. For example, the local microprocessor130 can check whether the user object is being moved into a wall in thegraphical environment; if so, the microprocessor can simply discard theposition data and report a constant position (or delta) to the hostcomputer that would cause the host to display the graphical objectagainst the wall surface. This allows the host computer to not beinvolved with the clipping process and simply display the graphicalobject at the reported position, thus freeing the host to perform othertasks.

The host computer 18 can also perform clipping in an embodiment. Sinceit is desirable to keep the application programs 202 and 204, theoperating system, and other high level programs ignorant of the clippingprocess, a lower level program preferably handles the clipping. Forexample, the translation layer 218 as shown in FIG. 4 (or alternativelythe context driver or API) can check for the conditions that causeclipping to be applied. If clipping is to be performed, the translationlayer alters the input position data appropriately before it is sent onto the context driver, API, operating system and any applicationprograms. Problems with this implementation include increased load onthe host with the overhead of intercepting and translating incomingmessages.

A different force feedback feature that can be implemented either by theforce feedback device or the host computer is “pressure clicks” or“click surfaces.” As described above, these are surfaces, objects orregions in a graphical environment which have additional functionalitybased on the position of the graphical object in relation to the object.Thus, a border of a window can be designated a click surface such thatif the user overcomes a resistive force and moves the graphical object(or user object) over a threshold distance into the border, a functionis activated. The function can be scrolling a document in the window,closing the window, expanding the window size, etc. A similar clicksurface can be used on an icon or button to select or activate the iconor button.

The force feedback device can implement click surfaces to easecomputational burden on the host computer. The local microprocessor cancheck whether the graphical object has moved into a click surface orregion and output the resistive force as appropriate. When the graphicalobject is moved past the threshold distance, the microprocessor reportsto the host computer that the action associated with the click surfacehas been made; for example, the microprocessor reports that a buttonpress or double click has been performed by the user. The host receivesa signal that a button has been pressed and acts accordingly. Or, themicroprocessor reports that a particular click surface has beenspecifically selected, where the host can interpret the selection andimplement an associated function.

The host may also implement at least a portion of the functionality ofclick surfaces. The microprocessor can output the resistive (or othertype) force associated with the click surface, but only reports the userobject position to the host. When the host determines that the graphicalobject (or user object) has moved beyond the threshold distance, thehost implements the function associated with the click surface(scrolling text, closing a window, etc.) As with clipping, it ispreferred that the translation layer handle this functionality bymodifying data accordingly before passing it to the context driver andAPI.

While the subject has been described in terms of embodiments, it iscontemplated that alterations, permutations and equivalents thereof willbecome apparent to those skilled in the art upon a reading of thespecification and study of the drawings. For example, although examplesin a GUI are described, the embodiments herein are also very well suitedfor other two-dimensional graphical environments and especiallythree-dimensional graphical environments, where a user would like finepositioning in manipulating 3-D objects and moving in a 3-D space. Forexample, the isometric limits are quite helpful to move a graphicalobject or controlled object in a 3-D environment further than physicallimits of the interface device allow. In addition, many different typesof forces can be applied to the user object 12 in accordance withdifferent graphical objects or regions appearing on the computer'sdisplay screen and which may be mouse-based force sensations orgraphical object-based force sensations. As stated above, many types ofuser objects and mechanisms can be provided to transmit the forces tothe user, such as a mouse, touch screen or other touch device,trackball, joystick, stylus, etc. Furthermore, certain terminology hasbeen used for the purposes of descriptive clarity, and are not limiting.

1. A system, comprising: a user device adapted to receive input from auser, the user device configured to provide input information based uponthe input received from the user associated with manipulation of agraphical object in a graphical environment; a processor configured todetermine a force effect to be output based on the input information,the user preference information further including information associatedwith at least one force effect from a plurality of force effects, eachforce effect from the plurality of force effects being associated with atype of graphical object from the plurality of types of graphicalobjects; a means for outputting a signal configured to cause the atleast one force effect to be output when a cursor displayed in thegraphical user interface interacts with a graphical object having afirst type from the at least one type, the graphical object beingassociated with an application program from a plurality of applicationprograms configured to run simultaneously, each application program fromthe plurality of application programs being configured to associate aforce effect from the plurality of force effects with a graphical objectfrom a plurality of graphical objects associated with that applicationprogram, each application program from the plurality of applicationprograms being configured to interact with a background applicationprogram, the background application program being configured toassociate a force effect from the plurality of force effects with atleast one application program from the plurality of applicationprograms.
 2. A method for simulating a multi-tasking graphicalenvironment implemented comprising: creating a context associated witheach of a plurality of application programs running in saidmulti-tasking graphical environment; receiving an input from a userinterface device; receiving force effect commands from said applicationprograms in response to said input; sending said force effect commandsin said context of an application program to a force feedback devicewhen said application program is active in said multi-taskingenvironment; commanding said force feedback interface device to output aforce effect in response to said force effect commands; and storing saidforce effect commands into said contexts, wherein each of said contextsis associated with at least one of said application programs running ona host computer.
 3. A method for generating a force feedback,comprising: determining a force feedback effect partially based oninformation associated with at least one type of graphical object from aplurality of types of graphical objects and information associated witha least one force feedback effect from a plurality of force feedbackeffects, the at least one type of graphical object being associated witha graphical user interface, each force feedback effect from theplurality of force feedback effects being associated with a type ofgraphical object from the plurality of types of graphic objects;outputting a signal configured to cause the at least one force feedbackeffect in response to an interaction between an input from a userinterface and a graphical object, the graphical object being associatedwith an application program from a plurality of application programsconfigured to run simultaneously, each application program from theplurality of application programs being configured to associate a forcefeedback effect from the plurality of force feedback effects with agraphical object from a plurality of graphical objects associated withthat application program, each application program from the plurality ofapplication programs providing information to a background applicationprogram.